A Practical Guide to GPU Monitoring for Silent Data Corruption (SDC)

Most GPU monitoring stacks were built to answer one question: is the card up? Utilization, power draw, HBM temperature, XID codes, ECC counters — the whole DCGM/nvidia-smi surface is oriented around liveness and thermal safety. That is table stakes for a rendering farm or an inference fleet. It is dangerously insufficient for a multi-thousand-GPU training cluster.

The failure mode that standard monitoring cannot see — and that dominates the unexplained interruption budget of every hyperscale training run published to date — is Silent Data Corruption (SDC). A gate-level defect in a single SM produces a numerically plausible but wrong result. No XID. No ECC increment. No thermal alarm. The gradient flows into All-Reduce and poisons the run.

This guide is for infrastructure and ML platform engineers who already run DCGM and are asking: what's the next layer, and why does it matter economically?

---

1. What conventional GPU monitoring actually catches

A typical production stack looks like this:

Every one of those signals is a fail-stop or fail-noisy signal. SDC is fail-silent by construction. The card reports healthy, the job reports healthy, and the loss curve reports healthy — for a while.

2. Why SDC is the dominant hidden-cost failure mode at scale

Three data points from the public record:

1. Meta's Silent Data Corruptions at Scale (Dixit et al., 2021) established that roughly 1 in 1,000 cores in a large fleet exhibits SDC-inducing defects. That base rate does not shrink; it multiplies with fleet size.
2. Meta's Llama 3 infrastructure report (2024) attributes 1.4% of unexpected interruptions on the 16k-GPU run to confirmed SDC — a lower bound, because SDC is under-attributed by definition.
3. Tung et al., The Anatomy of Silent Data Corruption (2026), showed via gate-level fault injection on production-class silicon that ~99% of corrupted outputs are numerically plausible (not NaN/±INF). Standard "value sanity" checks do not fire.

The economic translation is uncomfortable. A single undetected SDC event on a large run means: hours of wasted goodput across thousands of GPUs, a rollback to the last known-good checkpoint, and — if you did not catch it fast — a compromised model artifact that you may not be able to distinguish from a healthy one.

Standard monitoring's blind spot here is not a bug. It is an architectural mismatch: DCGM tells you a GPU is working; you need to know whether a GPU is correct.

3. The gap: continuous, in-band correctness

Research on SDC has bifurcated into two camps that both miss production reality:

• Redundant execution (DMR/TMR). Run every op twice or three times, vote. Detects almost everything. Costs 2–3× the capital. Nobody at scale runs this on training.
• Periodic out-of-band fleet testing. Take nodes offline, run known-answer kernels, quarantine anything that fails. Catches defects — but only between test windows, and forces the pipeline stalls it exists to prevent.

The missing layer is continuous, in-band correctness telemetry wired directly into the cluster's dispatch decisions. That is what ACE's Sticky SDC Detection provides, and it is why we treat it as a first-class primitive rather than an add-on to DCGM.

4. How Sticky SDC Detection works in ACE

The design constraints are strict:

• Zero training overhead. No shadow execution, no redundant math on the hot path.
• In-band. Signal must come from the actual training workload, not a synthetic probe.
• Sticky. A single anomalous step is a rumor; a pattern on a specific node is evidence. We refuse to act on rumors — see Storm Guards: When a Fault Detector Should Refuse to Act.
• Dispatch-aware. The output is not an alert; it is a derating signal the scheduler consumes.

The mechanism sits at the intersection of three data sources already flowing through the cluster:

1. Gradient / activation statistics at NCCL boundaries — distribution shifts that correlate with a specific rank rather than a step.
2. Cross-replica divergence — where data-parallel replicas should be numerically close, persistent per-rank drift is a strong SDC signal.
3. Historical per-node behavior — a node that has produced anomalous statistics across multiple unrelated jobs is a mercurial core, not a bad batch.

When those three agree, the node is derated in the catalog. The scheduler stops dispatching new training shards to it, in-flight work drains, and out-of-band diagnostics run on the quarantined host. The training run never sees the fault.

The mental model: DCGM watches the hardware. Sticky SDC watches the math.

5. What to instrument if you're building this yourself

If you are not using ACE and want to close the SDC gap in-house, here is the minimum viable telemetry:

• Per-rank gradient norms and activation-stat histograms, exported every step, tagged with (job_id, rank, node_id, gpu_uuid). Aggregation is done offline; the hot path only emits.
• Cross-replica divergence metrics for data-parallel groups — L2 distance between should-be-identical tensors at synchronization points.
• A per-node reputation store that survives across jobs. A single job's anomalies are noise; a node's career of anomalies is signal.
• Refusal logic on the action path. Do not derate on one flag. Require corroboration across time and across jobs. The cost of a false derate at 1000-GPU scale is real; the cost of a false quiet is worse.
• A dispatch integration. Detection without a scheduler hook is a dashboard. It is not a mitigation.

That last point is where most in-house efforts stall. Building the detector is a research problem. Wiring it into dispatch — safely, with storm guards — is a systems problem, and it is the reason we built ACE.

6. Where to go from here

If you want the theory, start with our deep-dive on the underlying failure mode: The Invisible Tax on AI Scale: Demystifying and Mitigating Silent Data Corruption in GPU Fleets.

If you want the operational counterpart — why a detector must sometimes refuse to act — read Storm Guards.

If you want to see Sticky SDC Detection running against your fleet, request alpha access.