ACEFLEET
ace-sidecar on GitHub
← /blog
· ACE Engineering#gpu #infra #reliability #sdc #cluster-ops #scorecard #in-house-gpu-fleet-stack

Mitigating Silent Data Corruption in Scaled AI Inference Fleets

Detecting and isolating gate-level GPU micro-faults in sub-5 seconds: preventing gradient poisoning and protecting Model FLOPs Utilization (MFU).

Training configurations regularly cross the multi-thousand-GPU threshold, and the operating metric for those clusters is Model Flops Utilization (MFU) and protective goodput. Silent Data Corruption (SDC) is one failure mode that reduces both without appearing in any log.

The Current Market Landscape

SDC moved from academic curiosity to documented infrastructure risk over roughly five years.

  • Foundational disclosures came from hyperscalers. Google's Cores That Don't Count (Hochschild et al., HotOS '21 [1]) coined the term mercurial cores — CPUs that miscompute silently — and Meta's Silent Data Corruptions at Scale (Dixit et al., 2021 [2]) put the rate on the order of 1 in 1,000 cores exhibiting SDC-inducing defects at fleet scale. Both establish SDC as a systemic property of large fleets rather than a manufacturing tail.
  • The GPU and LLM-training case is now documented. Understanding Silent Data Corruption in LLM Training (2025 [3]) and Exploring Silent Data Corruption as a Reliability Challenge in LLM Training (2026 [4]) trace how a single corrupt gradient propagates through All-Reduce across a run. LLM-PRISM (2026 [5]) characterizes SDC arising from permanent GPU faults, and The Anatomy of Silent Data Corruption (Tung et al., 2026 [6]) supplies gate-level fault-injection evidence on production-class silicon, finding NaN/±INF outcomes to be only ~1% of corruptions — the remainder are numerically plausible and therefore undetected by default.
  • Operational cost appears in flagship runs. Meta's Llama 3 report (2024 [7]) attributes 1.4% of unexpected interruptions (6 confirmed events in a 54-day snapshot) to SDC, and Google has publicly estimated an SDC event every week or two at Gemini scale.
  • Cross-vendor standardization is underway. The Open Compute Project's Silent Data Corruption in AI whitepaper [8] covers detection and reporting.

Production mitigations remain coarse. Redundant execution (DMR/TMR) is accurate at a 2–3× capital overhead. Periodic out-of-band fleet testing catches defects only between test windows, and forces the pipeline stalls it aims to prevent. Published research concentrates on detection and characterization; continuous, in-band isolation wired into the cluster's dispatch decisions is less covered.

Where ACE is different

Prior work either measures SDC offline (fault injection, trace characterization) or detects it with redundancy. ACE treats SDC-precursor isolation as a staged control loop co-located with economic dispatch: it correlates non-silent precursors (thermal drift, XID fault strings) into a health signal, prices the goodput a reclaim would risk, and graduates from passive SHADOW observation to human-approved ADVISORY to autonomous CLOSED_LOOP isolation, with hard capacity-floor storm guards so a correlated fault wave cannot self-starve the fleet.

1. What is Silent Data Corruption (SDC)?

Data center failures split into two categories:

  • Detected Unrecoverable Errors (DUEs): hardware faults that trigger machine-check exceptions, write to system logs, or throw explicit hardware flags (parity or instruction traps).
  • Silent Data Corruption (SDCs): hardware miscomputations that exit a processing unit with no hardware indication and no log footprint. The system continues, delivering corrupt values to applications.

Memory bit-flips are caught and mitigated by Error-Correcting Code (ECC) in High-Bandwidth Memory (HBM). SDCs bypass standard telemetry buffers entirely.

2. Why SDCs Exist

SDCs follow from physical and electrical limits being pushed simultaneously:

  • Shrinking process geometries. As transistors reach single-digit nanometer regimes, device margins tighten and the voltage boundary separating a logical 1 from a 0 narrows, raising vulnerability to transient and structural faults.
  • Power-Delivery Network (PDN) noise. High-density AI workloads draw extreme current and produce localized voltage droops. The resulting electrical noise creates timing violations across execution paths, causing arithmetic pipelines to latch values early or late.
  • Progressive wear-out and thermal degradation. Continuous operation produces localized thermal anomalies and electro-migration. Silicon that passes factory testing can degrade over months of intensive execution.
  • Unprotected logic paths. Memory arrays are shielded by ECC, but scheduling queues, register file multiplexers, and parts of the scalar and Tensor Core execution pipelines lack comprehensive structural checkers, so gate-level stuck-at faults manifest silently.

The Anatomy of Silent Data Corruption: GPU Error Pattern Study and Modeling Guidance (Tung et al., 2026) ran gate-level stuck-at fault injection on a production-class data-center GPU across 63 CUDA micro-benchmarks, measuring how architectural vulnerabilities and workload profiles interact to produce non-deterministic mathematical errors within execution pipelines.

3. The Cost to Training and Inference

Corrupted gradient variance

During synchronous distributed training, thousands of GPUs calculate gradients in parallel and average them across an All-Reduce collective. A single faulty GPU introducing a silent bit-flip or numerical nullification into a weight tensor propagates that value across the cluster.

If the corruption does not trigger a Not-a-Number (NaN) trap, it manifests as corrupted gradient variance. Infrastructure monitoring reports the run as healthy — nodes report healthy state, logs stay clear — while the optimization is degraded. The run can continue for days into local minima or gradient explosions before the condition is identified.

Scale multiplier

Meta reported that SDCs accounted for 1.4% of unexpected GPU interruptions during Llama 3 pre-training (6 confirmed SDC interruptions in a 54-day snapshot), and Google has estimated an SDC event roughly every week or two during Gemini-scale training. Because a single corrupt tensor propagates across the entire All-Reduce ring, one un-isolated SDC affects a full synchronous step, so the cost is fleet-wide rather than proportional to one GPU, and accrues until the condition is detected.

4. How the Industry Detects SDCs Today

Two paradigms are in production use:

  1. Dual and Triple Modular Redundancy (DMR/TMR): running identical workloads across multiple accelerators and comparing outputs. Accurate, at 2× to 3× capital overhead.
  2. Periodic out-of-band functional testing: stopping the cluster every few hours to run dedicated mathematical stress tests such as CUDA micro-benchmark suites or structural gate-level simulations. This adds orchestration overhead, forces pipeline restarts, and leaves a vulnerability window between test cycles.

5. The ACE Approach

ACE (AI Compute Efficiency) integrates SDC detection and isolation into the live operational fabric of the cluster, decoupling the analytical core from the cluster runtime and executing a multi-tier protective loop without stopping production work.

Fast-loop telemetry

ACE deploys a lightweight host-native daemon that hooks into localized hardware layers. A real-time telemetry engine processes sub-surface signals, identifying statistical micro-deviations — localized thermal anomalies and correlated hardware fault codes — that act as precursors to physical gate degradation. The corruption is silent; these precursor signals are not, which is what allows a drifting accelerator to be isolated before it affects a training step.

Multi-tier deployment

  • Tier 1: SHADOW mode (passive ingestion). ACE runs read-only. It strips sensitive workload metadata, monitors fleet hardware noise passively, and maps historical fault patterns against a live Fleet-Efficiency Scorecard, exposing where compute hours are being lost without any mutation.
  • Tier 2: ADVISORY mode (human-in-the-loop). When the ACE Brain flags a node exhibiting high-probability SDC signatures, it compiles a diagnostic payload and notifies on-call infrastructure engineers via Slack or PagerDuty. An interactive approval control isolates the node.
  • Tier 3: CLOSED_LOOP mode (autonomous). A configuration toggle activates full automation. ACE acts as an autonomous backend for orchestrators such as Slurm or Kubernetes (via native SPANK plugins or custom controllers), executing millisecond-level node drains or taints to isolate degraded hardware.

Storm guards

Local Storm Guards run entirely on the cluster perimeter. Rather than a fixed drain quota, ACE gates every hard eviction on how much healthy capacity it would leave: it hard-evicts when plenty remains, derates (keeps the node serving at reduced trust) in the ambiguous middle zone, and halts the eviction entirely — holding the node at full capacity and paging an operator — once the fleet approaches an emergency capacity floor. A correlated fault storm therefore cannot drive the cluster below that floor.

The pattern is the same shadow-scorecard discipline ACE applies elsewhere in the stack, where the fast loop recovers roughly a fifth of avoidable cost: measure the counterfactual first, then close the loop.

Further Reading

Open Compute Project whitepaper Silent Data Corruption in AI (Nishant George, NVIDIA), and the gate-level fault-injection study The Anatomy of Silent Data Corruption.

References

  1. P. H. Hochschild, P. Turner, J. C. Mogul, et al. Cores That Don't Count. HotOS 2021. research.google
  2. H. D. Dixit, S. Pendharkar, M. Beadon, et al. Silent Data Corruptions at Scale. 2021. arXiv:2102.11245
  3. Understanding Silent Data Corruption in LLM Training. 2025. arXiv:2502.12340
  4. Exploring Silent Data Corruption as a Reliability Challenge in LLM Training. 2026. arXiv:2604.00726
  5. LLM-PRISM: Characterizing Silent Data Corruption from Permanent GPU Faults in LLM Training. 2026. arXiv:2604.10390
  6. C.-H. Tung, Y. Huang, N. Saxena, et al. The Anatomy of Silent Data Corruption: GPU Error Pattern Study and Modeling Guidance. 2026. arXiv:2605.04213
  7. Llama Team, AI @ Meta. The Llama 3 Herd of Models. 2024. arXiv:2407.21783
  8. N. George (NVIDIA). Silent Data Corruption in AI. Open Compute Project whitepaper. opencompute.org

Sign up to ACE now