Traffic Storm Protection: SLA Preservation and Concurrency Throttling

Consider 400 GPUs failing in the same second — a top-of-rack switch flaps, a power domain browns out, a NIC firmware bug fires across a SKU — and a correct fault detector responding to each one correctly: evict, evict, evict. Every individual decision is right. The aggregate cordons a third of the fleet, the surviving nodes inherit the load, saturate, fail their own health checks, and become eviction candidates in turn.

A fault detector that can only act is an amplifier. The capability that distinguishes a reliability control loop from one is the ability to identify a fault correctly and decline to act on it.

Prior art

This is one of the best-documented failure classes in distributed systems.

• Metastable failures. Bronson et al. named the pattern at HotOS '21 [1]: a system pushed past a tipping point by a trigger, then held in a degraded state by a sustaining feedback loop after the trigger is gone. Huang et al., Metastable Failures in the Wild, OSDI '22 [2], studied 22 such failures across 11 organizations and reported that a large share of major cloud outages fit the shape. The sustaining loop is usually work amplification under load; automated retries and automated evictions are both amplifiers.
• Retry and load amplification. Dean & Barroso, The Tail at Scale [3], documents how rare per-node hiccups become fleet-wide latency under fan-out, and how naive mitigation — retry everything, hedge everything — doubles load precisely under overload.
• Load shedding as a first-class control. WeChat's DAGOR [4], five years in production, is the reference design: priority-aware, collaborative shedding driven by real-time queue pressure rather than treating every request as mandatory.
• Circuit breakers. Nygard's Release It! [5] wraps a failing dependency in a state machine that trips open and stops sending it work, so one sick component cannot drain the pools keeping healthy ones alive.
• GPU fleets specifically. Meta's Llama 3 report [6] recorded 419 unexpected interruptions over a 54-day snapshot, ~78% traced to hardware with GPUs the single largest category, while holding >90% effective training time. Manual intervention was needed three times.

The common conclusion across the five: at scale, the costliest failures are the ones the operator's own automation sustains, and the mitigation is a governor on the control loop rather than a better detector.

ACE's design

Most fault handlers are a chain of conditionals: this signal → evict, that signal → quarantine. ACE makes the response a function of fleet state instead. Two principles:

1. Every hard eviction is gated on the healthy capacity it would leave behind. Plenty of capacity remaining: evict the anomaly. Getting tight: derate instead of cut. At the emergency floor: halt the eviction and page a human. A correlated storm therefore cannot drive the fleet below its floor, because the last evictions before the floor refuse themselves.
2. Fault semantics are a policy table, not a code path. What a signal means is data — a lookup from hardware signal to response class — kept separate from the storm governor that decides whether the fleet can afford that response.

The governor: evict, derate, halt

A classifier scores every proposed hard eviction by the healthy finite-capacity fraction that would remain after shedding the node, and resolves to one of three tiers:

• Evict — plenty of healthy capacity remains. Treated as a single-node anomaly and hard-evicted.
• Derate — enough faults are landing that shedding is starting to bite, but the fleet is not at the floor. The node is pinned into a derated tier: still in service, still absorbing load, no longer trusted with everything.
• Halt — the fleet has hit its emergency floor. The eviction is not applied. The node is held at full capacity and an alert is raised. The system records the fault as real and selects raw availability over correctness, because the alternative is self-starvation.

The floor holds strictly: because a halt leaves capacity untouched, no sequence of faults can push the healthy fraction below the floor through eviction — the evictions that would cross it are the ones that refuse to fire. Only finite-capacity destinations count toward the fraction; an elastic pay-as-you-go overflow tier counts as headroom, not fleet floor.

Every intermediate outcome — a derate, a halted eviction — is a first-class alertable event rather than a silently dropped action. The governor is opt-in, with a floor the fleet operator chooses.

The other two governors: retries and hedges

The classic metastable loop is a retry storm, so the request path is budgeted the same way.

• A fleet-global retry budget caps total retries across a run as a fraction of total requests. A correlated brown-out makes every request fail; without a global cap, every failure retries and the retries become the load. The budget makes retrying a scarce shared resource rather than a per-request right.
• A hedge budget does the same for tail-latency hedging. A hedge fires a second parallel request when the primary misses its latency target; under a brown-out every request misses it, and unbounded hedging doubles fleet load. The budget caps cumulative hedges as a fraction of requests seen.

Both budgets can be disabled in a controlled test, so the amplification storm can be reproduced deliberately and compared against the governed case.

Not every fault means evict

The governor decides whether the fleet can afford a response. A separate stateless policy table decides what a signal deserves. The hardware-fault classifier maps known hardware signals to a small set of response classes — cordon-for-diagnostics, hard-evict, interconnect-evict — by lookup rather than branching logic.

The entries left out are the informative ones. A clean, unambiguous hardware-reboot signal maps to a hard evict. A memory-page fault does not, because that signal is most often an application's illegal memory access rather than a sick GPU, and auto-evicting a healthy node on every bad pointer dereference is itself a self-inflicted storm. Treating that signal as hardware requires a rate-aware rule — several within a window, not one — which belongs in a stateful detector, not a stateless lookup that fires on a single event.

Flap escalation

Refusing to act is not the same as being passive. The fleet health intake also runs a flap guard pointed the other direction: a node that fast-evicts on a transient glitch, passes its reboot check, rejoins the pool, and then crashes the next job on the same edge case. Individually each trip is auto-recoverable; in aggregate the node is a serial job-killer. Once a destination accumulates enough fast-evict trips inside a rolling window, ACE escalates the next trip from an auto-recoverable eviction to a sticky quarantine — cordoned for diagnostics, readmitted only by an explicit human clear.

The governor therefore operates in both directions: how hard to respond is a function of evidence and fleet state, rather than one fixed action per signal.

---

References

1. N. Bronson, A. Aghayev, A. Charapko, T. Zhu. Metastable Failures in Distributed Systems. HotOS 2021. dl.acm.org
2. L. Huang, M. Magnusson, A. B. Muralikrishna, et al. Metastable Failures in the Wild. USENIX OSDI 2022. usenix.org
3. J. Dean, L. A. Barroso. The Tail at Scale. Communications of the ACM, 56(2), 2013. research.google
4. H. Zhou, M. Chen, Q. Lin, et al. Overload Control for Scaling WeChat Microservices. ACM SoCC 2018. arXiv:1806.04075
5. M. T. Nygard. Release It!: Design and Deploy Production-Ready Software. Pragmatic Bookshelf, 2007 (2nd ed. 2018) — origin of the Circuit Breaker stability pattern. pragprog.com
6. Llama Team, AI @ Meta. The Llama 3 Herd of Models. 2024. arXiv:2407.21783

---

Sign up to ACE now