Detecting GPU Silent Data Corruption: Autonomous Health Probes and Automatic Isolation
How standard DCGM/XID GPU monitoring misses Silent Data Corruption (SDC) in AI clusters, and how ACE's continuous validation engine isolates corrupt nodes in under 5 seconds.
Most GPU monitoring stacks answer one question: is the card up? Utilization, power draw, HBM temperature, XID codes, ECC counters — the DCGM/nvidia-smi surface is oriented around liveness and thermal safety.
Silent Data Corruption (SDC) is outside that surface. A gate-level defect in a single SM produces a numerically plausible but wrong result: no XID, no ECC increment, no thermal alarm. The gradient flows into All-Reduce and propagates through the run. This guide covers what a DCGM-based stack misses and what layer sits above it.
1. What conventional GPU monitoring catches
| Signal? | What it detects? | What it misses? |
|---|---|---|
nvidia-smi / DCGM utilization |
Idle GPUs, workload imbalance | Corrupt math on a busy GPU |
| XID error codes | Driver / hardware faults that halt execution | Faults that return a wrong number and keep going |
| ECC counters | Detected & corrected memory bit-flips | SM/ALU miscompute (no memory involved) |
| Power & thermals | Throttling, PSU issues | Logic-path defects that are temperature-independent |
| NCCL timeouts | Rank stalls, network partitions | Nodes that finish on time with wrong values |
Each of those is a fail-stop or fail-noisy signal. SDC is fail-silent by construction: the card reports healthy, the job reports healthy, and the loss curve reports healthy.
2. Measured scale of the problem
Three data points from the public record:
- Meta's Silent Data Corruptions at Scale (Dixit et al., 2021) established that roughly 1 in 1,000 cores in a large fleet exhibits SDC-inducing defects. That base rate multiplies with fleet size.
- Meta's Llama 3 infrastructure report (2024) attributes 1.4% of unexpected interruptions on the 16k-GPU run to confirmed SDC — a lower bound, since SDC is under-attributed by definition.
- Tung et al., The Anatomy of Silent Data Corruption (2026), showed via gate-level fault injection on production-class silicon that ~99% of corrupted outputs are numerically plausible (not NaN/±INF), so value-sanity checks do not fire.
An undetected SDC event on a large run costs goodput across thousands of GPUs, a rollback to the last known-good checkpoint, and a model artifact whose validity cannot be distinguished from a healthy one without further work.
The blind spot is an architectural mismatch rather than a defect: DCGM reports whether a GPU is working, not whether it is correct.
3. The gap: continuous, in-band correctness
SDC research divides into two approaches:
- Redundant execution (DMR/TMR). Run every op twice or three times and vote. Detects almost everything at 2–3× the capital cost.
- Periodic out-of-band fleet testing. Take nodes offline, run known-answer kernels, quarantine failures. Catches defects only between test windows, and forces the pipeline stalls it exists to prevent.
The layer neither covers is continuous, in-band correctness telemetry wired into the cluster's dispatch decisions. That is what ACE's Sticky SDC Detection provides.
4. How Sticky SDC Detection works
Design constraints:
- Zero training overhead. No shadow execution, no redundant math on the hot path.
- In-band. Signal comes from the actual training workload, not a synthetic probe.
- Sticky. A single anomalous step is not evidence; a pattern on a specific node is. See Storm guards.
- Dispatch-aware. The output is a derating signal the scheduler consumes, not an alert.
The mechanism combines three data sources already flowing through the cluster:
- Gradient / activation statistics at NCCL boundaries — distribution shifts that correlate with a specific rank rather than a step.
- Cross-replica divergence — where data-parallel replicas should be numerically close, persistent per-rank drift is an SDC signal.
- Historical per-node behavior — a node producing anomalous statistics across multiple unrelated jobs indicates a mercurial core rather than a bad batch.
When those three agree, the node is derated in the catalog. The scheduler stops dispatching new training shards to it, in-flight work drains, and out-of-band diagnostics run on the quarantined host. The training run does not see the fault.
Stated compactly: DCGM watches the hardware; Sticky SDC watches the math.
5. Minimum telemetry for an in-house implementation
- Per-rank gradient norms and activation-stat histograms, exported every step, tagged with
(job_id, rank, node_id, gpu_uuid). Aggregation runs offline; the hot path only emits. - Cross-replica divergence metrics for data-parallel groups — L2 distance between should-be-identical tensors at synchronization points.
- A per-node reputation store that survives across jobs. A single job's anomalies are noise; a node's record across jobs is signal.
- Refusal logic on the action path: corroboration across time and across jobs before a derate. A false derate at 1000-GPU scale has a measurable cost, as does a false negative.
- A dispatch integration. Detection without a scheduler hook produces a dashboard, not a mitigation.
Building the detector is a research problem; wiring it into dispatch with storm guards is a systems problem.
6. Related reading
- Silent data corruption in GPU fleets — the underlying failure mode.
- Storm guards — why a detector must sometimes decline to act.