SDC detection
Canary matrix multiplication checks for silent data corruption on GPU nodes.
problem it solves
Stops hardware numerical calculation errors from producing silent wrong outputs.
What it does
What it does: Evaluates real-time token stream mathematical invariants, dual-node shadow consensus, and background deterministic canary matrix multiplications to detect silent data corruption.
Inference vs Training SDC detection: At inference time (where no backward pass or training loss exists), SDC detection evaluates 4 real-time stream invariants: non-finite (NaN/±Inf) logit checks, logit distribution entropy bounds, dual-node shadow consensus verification (cosine distance divergence > 0.15 on deterministic prompts), and background canary matmul checksums. For training/fine-tuning clusters, it additionally monitors loss explosion surges (>4x running min) and per-rank gradient variance spikes.
Cloud APIs vs Self-Hosted / Bare-Metal GPUs: On managed cloud APIs (OpenAI, Bedrock, Anthropic, etc.), SDC detection inspects egress token streams for mathematical invariants (catching non-finite NaN/±Inf logits and entropy collapse) and dual-provider consensus in under 2 µs. Full hardware-level protections—including rolling canary GEMM matmuls on idle cards, physical node drainage, sticky quarantine, and 3-cycle automated diagnostic probe recovery—require self-hosted or bare-metal GPU clusters (vLLM, DGX/HGX, Kubernetes).
Why it is 'Ready now' without GPU fleet registration: Hot-path stream invariant inspection runs directly inside the gateway proxy layer on any configured model route without requiring physical GPU registration. Preconditions for host-level telemetry are advisory (non-blocking) so teams can activate gateway stream inspection immediately in Shadow or Prod mode.
What it watches: Hot-path non-finite logit violations, dual-node shadow consensus divergence, and bit-level numerical drift in background canary GEMMs.
When it triggers: Real-time on every egress token stream, plus background rolling schedules across active fleet GPU nodes.
The Action: Ejects and drains degraded hardware nodes into sticky quarantine before corrupt outputs reach production applications or downstream agent loops.
How it recovers: Quarantined nodes remain off-line until out-of-band diagnostics pass 3 consecutive recovery cycles.
What we need from you
- Configured models or routesrequired
At least one model route or provider enabled to inspect egress token streams for mathematical invariants.
- ACE node agent on GPU hosts (for hardware quarantine)recommended
Executes low-level canary matmuls on Tensor Cores and streams hardware invariants to the gateway. Required for physical node isolation, optional for cloud API stream inspection.
- Canary execution budgetrecommended
Allocates <0.5% of node compute hours for background checks on owned GPU clusters.
What each mode does
| Mode | Effect on your request | What you can see |
|---|---|---|
| off | Not consulted. GPUs run without silent data corruption validation. | No sdc_detection stage recorded. |
| shadow | Runs canary checks and logs detected drift without ejecting faulted hardware. | Stage with action=would_eject_sdc and error_bits logged. |
| prod | Ejects and quarantines GPUs exhibiting numerical drift automatically. | Quarantined GPU ID and canary checksum mismatch logged. |
Current policy
| Check frequency | Rolling 15-minute schedule | Sub-0.5% compute overhead budget. |
| Verification method | Deterministic GEMM checksum comparison | Bit-exact reference validation. |
| Ejection action | Immediate drain & node quarantine | Prevents corrupt output generation. |
Worth knowing before you enable it
- ·Silent Data Corruption (SDC) produces mathematically incorrect answers without raising hardware exceptions or DCGM XID error codes.
- ·Cloud APIs support gateway-level stream invariants (NaN/Inf logits, entropy collapse), but physical node quarantine and canary GEMMs require self-hosted/owned GPU hosts.
- ·SDC detection shows as 'Ready now' on cloud API accounts because gateway-level streaming checks require no physical cluster registration and host telemetry preconditions are non-blocking.
- ·At inference time, SDC surfaces as corrupted agent steps, malformed JSON, or NaN/±Inf logits rather than training loss explosions.
- ·Overclocked or thermal-throttled GPUs exhibit higher rates of bit flip drift.
- ·Canary tests run in idle gaps between active inference requests.
What it replaces
- ·Silent wrong answers in LLM reasoning outputs.
- ·Corrupted training checkpoint weights.
- ·Manual GPU burn-in testing runbooks.
Deep hardware diagnostic suites and automated vendor RMA ticketing on enterprise.
- ·Custom CUDA kernel stress test canary suites.
- ·Automated cloud vendor RMA node replacement ticketing.
- ·Historical GPU bit-drift reliability scoring.