Storm guards: why a fault detector has to be able to refuse to act

Here is the failure mode that keeps me up at night, and it is not a GPU dying.

It's 400 GPUs dying at the same second — a top-of-rack switch flaps, a power domain browns out, a NIC firmware bug fires across a whole SKU — and a perfectly correct fault detector responds to each one perfectly correctly: evict, evict, evict. Every individual decision is right. The aggregate is that your reliability system just cordoned a third of the fleet, the surviving nodes inherit the load, they saturate, their health checks start failing, and now the evictor is coming for them too. The cure has become the outage.

The uncomfortable lesson: a fault detector that can only ever act is not a safety system. It's an amplifier. The single most important capability a reliability control loop has is the ability to look at a fault it correctly identified and decide not to act on it — because acting would make things worse.

The current market landscape

This isn't a niche concern; it's one of the best-documented failure classes in distributed systems, and it has a name.

• Metastable failures. Bronson et al. named the pattern at HotOS '21 [1]: a system pushed past a tipping point by a trigger, then held in a degraded state by a sustaining feedback loop even after the trigger is gone. Huang et al. followed with Metastable Failures in the Wild at OSDI '22 [2], studying 22 such failures across 11 organizations and noting that a large share of major cloud outages fit the shape. The sustaining loop is almost always work amplification under load — and automated retries or automated evictions are textbook amplifiers.
• Retry and load amplification. Dean & Barroso's The Tail at Scale [3] is the canonical treatment of how rare per-node hiccups become fleet-wide latency once you fan out — and how naive mitigation (retry everything, hedge everything) doubles load exactly when you can least afford it.
• Load shedding as a first-class control. Production systems that survive overload do so by refusing work on purpose. WeChat's DAGOR [4] — five years in production — is the reference design: priority-aware, collaborative shedding driven by real-time queue pressure, not by treating every request as sacred.
• Circuit breakers. The stability pattern behind all of this comes from Nygard's Release It! [5]: wrap a failing dependency in a state machine that trips open and stops sending it work, so one sick component can't drain the pools that keep the healthy ones alive.
• And it's now a GPU-fleet problem. Meta's Llama 3 report [6] recorded 419 unexpected interruptions over a 54-day snapshot — ~78% traced to hardware, GPUs the single largest category — while still holding

  90% effective training time. The thing that made that number survivable was automation that knew when not to fire: manual intervention was needed only three times.

Put together, the literature says the same thing from five directions: at scale, the failures that hurt most are the ones your own automation sustains. The mitigation is not a better detector. It's a control loop with a governor on it.

Where ACE is headed

Most fault handlers are a chain of ifs: this signal → evict, that signal → quarantine. ACE is built around a different idea entirely — the fleet's own response to a fault should bend to the fleet's own state, not fire the same way regardless of context. Two principles anchor that:

1. Every hard eviction is gated on the healthy capacity it would leave behind. The response to a fault is not a fixed action — it's a function of fleet state. Plenty of capacity left? Evict the anomaly. Getting tight? Derate instead of cut. At the emergency floor? Halt the eviction entirely and page a human. A correlated storm therefore cannot drive the fleet below its floor, because the last evictions before the floor refuse themselves.
2. Fault semantics are a policy table, not a code path. What a signal means is data — a lookup from hardware signal to response class — kept separate from the storm governor that decides whether the fleet can currently afford that response.

The governor: hard-evict → derate → halt

At the center of ACE's storm guard is a classifier that scores every proposed hard eviction by the healthy finite-capacity fraction that would remain after shedding the node, and resolves to one of three tiers:

• Evict — plenty of healthy capacity remains. This is almost certainly a single-node anomaly, so it's hard-evicted normally.
• Derate — the ambiguous middle: enough faults are landing that shedding is starting to bite, but the fleet isn't in crisis. Instead of cutting the node, ACE pins it into a derated tier — kept in service at reduced trust, still absorbing load, no longer trusted with everything.
• Halt — the fleet has hit its emergency floor. The eviction is not applied. The node is held at full capacity, and an alert goes to a human. The system saw the fault, agreed it was real, and chose raw availability over correctness because the alternative was self-starvation.

The floor is designed to hold strictly: because a halt leaves capacity untouched, no sequence of faults can push the healthy fraction below the floor through eviction — the evictions that would cross it are the exact ones that refuse to fire. Only finite-capacity destinations count toward that fraction; an elastic, pay-as-you-go overflow tier is headroom, not fleet floor.

Every intermediate outcome — a derate, a halted eviction — is a first-class, alertable event. "The system chose to hold" is never invisible; it's a logged decision, not a silently dropped action. And the governor is something a fleet opts into with a floor it chooses, not a hidden behavior that surprises anyone the first time a rack drops.

The other two governors: retries and hedges

Eviction is the storm guard everyone forgets, but ACE treats the request path the same way, because the classic metastable loop is a retry storm.

• A fleet-global retry budget caps total retries across a run as a fraction of total requests. A correlated brown-out makes every request fail; without a global cap, every failure retries, and the retries are now the load. The budget makes retrying a scarce, shared resource rather than a per-request right — the same insight DAGOR applies to load shedding, applied here to retries.
• A hedge budget does the same for tail-latency hedging. A hedge fires a second parallel request when the primary misses its latency target — wonderful until a brown-out makes every request miss it, at which point unbounded hedging doubles fleet load at the worst possible moment. The budget caps cumulative hedges as a fraction of requests seen.

Both budgets can be turned off entirely in a controlled test, precisely so you can reproduce the amplification storm on purpose and watch the governor earn its keep by comparison.

Not every fault means evict

The governor decides whether the fleet can afford a response. A separate, stateless policy table decides what a signal even deserves. The hardware-fault classifier is deliberately boring data: known hardware signals map to a small set of response classes — cordon-for-diagnostics, hard-evict, interconnect-evict — by lookup, not by branching logic.

The interesting entries are the ones deliberately left out. A clean, unambiguous hardware-reboot signal maps straight to a hard evict. A memory-page fault does not — because that signal is most often an application's illegal memory access, not a sick GPU, and auto-evicting a healthy node every time a user's kernel dereferences a bad pointer is its own kind of self-inflicted storm. Treating that signal as hardware requires a rate-aware rule — several within a window, not one — and that belongs in a stateful detector, not a stateless lookup that fires on a single event.

That restraint — a fault you can see, correctly identify, and still choose not to act on — is the same principle as the storm floor, one layer down.

Flap escalation: the exception that proves the rule

Refusing to act is not the same as being passive. The fleet health intake also runs a flap guard pointed the other direction: a node that fast-evicts on a transient glitch, passes its reboot check, rejoins the pool, and promptly crashes the next job on the same edge case. Individually each trip is auto-recoverable; in aggregate the node is a serial job-killer. Once a destination accumulates enough fast-evict trips inside a rolling window, ACE escalates the next trip from an auto-recoverable eviction to a sticky quarantine — cordoned for diagnostics, readmitted only by an explicit human clear.

So the governor cuts both ways, and that symmetry is the whole philosophy: the system decides how hard to respond based on evidence and fleet state, rather than reflexively applying one fixed action per signal. Hold the storm; cordon the repeat offender. Both are the control loop refusing to be a simple if.

The takeaway

A reliability system is judged on its worst hour, not its average one, and its worst hour is a correlated storm. In that hour, the detector that fires on every true fault is the one that finishes the outage the switch started. The system that survives is the one that can hold a correct decision in its hand and say not now — governed retries, governed hedges, and above all an eviction that gates itself on the capacity floor it must never cross.

Build the governor before you build the automation it governs. The order matters, because you only find out you needed it during the one hour you can't afford to learn.

---

References

1. N. Bronson, A. Aghayev, A. Charapko, T. Zhu. Metastable Failures in Distributed Systems. HotOS 2021. dl.acm.org
2. L. Huang, M. Magnusson, A. B. Muralikrishna, et al. Metastable Failures in the Wild. USENIX OSDI 2022. usenix.org
3. J. Dean, L. A. Barroso. The Tail at Scale. Communications of the ACM, 56(2), 2013. research.google
4. H. Zhou, M. Chen, Q. Lin, et al. Overload Control for Scaling WeChat Microservices. ACM SoCC 2018. arXiv:1806.04075
5. M. T. Nygard. Release It!: Design and Deploy Production-Ready Software. Pragmatic Bookshelf, 2007 (2nd ed. 2018) — origin of the Circuit Breaker stability pattern. pragprog.com
6. Llama Team, AI @ Meta. The Llama 3 Herd of Models. 2024. arXiv:2407.21783