Modeling the failure process you can't afford to reproduce

Here is the uncomfortable thing about recovery policy for large-scale training: the regime you most need to test is the one you can least afford to produce.

At 1,024 GPUs a job's mean time to failure is already down to about 7.9 hours [1]. Meta's own projection puts a 16,384-GPU job at 1.8 hours and a 131,072-GPU job at 0.23 hours — a failure every fourteen minutes. You do not get to deliberately kill a run at that scale to see whether your checkpoint cadence, your re-placement logic, and your preemption budget actually hold up. The run is the experiment, and it costs six figures a day.

So the failure process has to be modeled. And the moment you decide to model it, the interface to that model becomes the product — because the mechanism is generic and reusable, and the only thing that is ever truly specific to your fleet is the data.

The current market landscape

Two facts are now well established in the literature, and they point in opposite directions.

Failures at scale are frequent, correlated, and dominated by hardware. Meta's Revisiting Reliability in Large-Scale ML Research Clusters (Meta, HPCA 2025 [1]) analyzed 11 months across two clusters — 4 million jobs, 150M+ A100 GPU-hours — and found MTTF collapsing roughly two orders of magnitude between an 8-GPU job (47.7 days) and a 1,024-GPU job (7.9 hours). The Llama 3 report (Meta, 2024 [2]) is consistent: over a 54-day pre-training snapshot it saw 419 unexpected interruptions, ~78% traced to hardware. The Philly trace study (Jeon et al., ATC '19 [3]) and the Acme LLM- development characterization (Hu et al., NSDI '24 [4]) both dedicate whole sections to failure as a first-class property of the cluster, not an exception path.

Recovery is expensive, and its efficiency is a tunable, not a constant. The classic result is Young's first-order [5] and Daly's higher-order [6] optimal-checkpoint-interval formula — checkpoint too often and you drown in write stalls, too rarely and a fail-stop rolls back too much work; the optimum scales as roughly √(2 · δ · MTBF). CheckFreq (Mohan et al., FAST '21 [7]) drove recovery from hours to seconds by tuning checkpoint frequency online within a 3.5% overhead bound. Systems like Bamboo (Thorpe et al., NSDI '23 [8]) and Oobleck (Jang et al., SOSP '23 [9]) go further — redundant pipeline computation and pipeline- template replication to survive failures without a full restart.

The tension: failures are non-memoryless and heavy-tailed in practice, but almost every recovery-policy evaluation still assumes a fixed exponential MTBF — a single mean, memoryless, one failure indistinguishable from the next. That assumption is convenient and wrong, and it is wrong in the direction that makes your policy look better than it is.

Where ACE is different

ACE does not try to own checkpoint mechanics — those stay framework-owned. What it owns is the economic response to failure — gang re-placement, checkpoint-aware preemption, node quarantine — driven by a model of the failure process built on empirical distributions rather than a textbook average. Instead of assuming one fixed mean-time-between-failures, the model draws from the fleet's own historical failure and checkpoint-latency patterns, so the picture it produces is genuinely non-memoryless: it can represent a cluster that is healthy for days and then thrashes every twenty minutes, which a single exponential mean simply cannot capture. A fleet's own failure history becomes a direct, swappable input to the model, not a parameter to guess at.

What gets modeled

Two empirical distributions drive the picture, in place of a single scalar mean-time-to-failure: historical inter-failure timing, and measured per-checkpoint save/load latency. Both are resampled per event, which is the single change that turns the fail-stop process from memoryless into a realistic one — a bimodal failure pattern produces failure timing a flat exponential simply cannot represent.

Every training job in the model carries a fail-stop / checkpoint lifecycle: a failure drawn from the distribution (or, absent one, a scale-aware estimate calibrated to the ≈7.9-hour-at-1,024-GPU figure from the literature [1], so a large gang fails far more often than a single card), each fail-stop rolling the job back to its last checkpoint and losing the progress since. It's purely observational by design — it never alters routing, cost, or served goodput on its own — and it's fully deterministic and replayable, so a recovery policy can be evaluated against the exact same failure trace as many times as it takes to trust the answer.

The metric that falls out: ETTR

The headline number is ETTR — Effective Training Time Ratio: productive runtime over wall-clock time, the same construction the industry uses to express training goodput as a function of job parameters [1]. It decomposes the unproductive remainder into:

• overhead / init — scheduling barrier, container/driver init, dataset and checkpoint staging;
• wasted progress — compute since the last checkpoint lost to a fail-stop rollback;
• checkpoint-write stall — the synchronous cost of dumping state.

Alongside that decomposition sits the expected wasted-progress cost implied by the checkpoint cadence itself (the Young/Daly tradeoff [5][6] made explicit), the scale-aware failure-rate estimate, the GPU-hours lost, and a second-order term for the churn a fail-stop induces in neighboring jobs sharing the same fleet. All of it rolls up into one fleet-level view.

Why the distribution matters — the whole point in one result

Feeding the model a frequent-failure distribution versus a rare one drives end-to-end ETTR from roughly 0.94 down toward 0.00. That spread is the argument for the whole exercise: a fixed exponential collapses two very different fleets into one average and hides exactly the regime — the frequently-failing, heavy-tailed one — where a recovery policy either earns its keep or falls over.

The takeaway

You cannot fuzz a 16K-GPU training run by breaking it on purpose. What you can do is make the failure process a first-class, swappable input — sampled from real history, non-memoryless, deterministic to replay — and validate a recovery policy against it before betting a real run on it. Build the mechanism, expose the distribution as the interface, and let every fleet supply the one thing only it can: its own failure history. That's the direction this is headed — recovery policy that's proven against your fleet's actual failure shape before it ever has to survive it live.

---

References

1. Meta / AI at Meta. Revisiting Reliability in Large-Scale Machine Learning Research Clusters. IEEE HPCA 2025. arXiv:2410.21680
2. Llama Team, AI @ Meta. The Llama 3 Herd of Models. 2024. arXiv:2407.21783
3. M. Jeon, S. Venkataraman, A. Phanishayee, et al. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758
4. Q. Hu, Z. Ye, Z. Wang, et al. Characterization of Large Language Model Development in the Datacenter. USENIX NSDI 2024. arXiv:2403.07648
5. J. W. Young. A First Order Approximation to the Optimum Checkpoint Interval. Communications of the ACM, 1974. (See also Checkpointing à la Young/Daly: An Overview, ACM DOI:10.1145/3549206.3549328)
6. J. T. Daly. A Higher Order Estimate of the Optimum Checkpoint Interval for Restart Dumps. Future Generation Computer Systems, 2006. DOI:10.1016/j.future.2004.11.016
7. J. Mohan, A. Phanishayee, V. Chidambaram. CheckFreq: Frequent, Fine-Grained DNN Checkpointing. USENIX FAST 2021. usenix.org
8. J. Thorpe, P. Zhao, J. Eyolfson, et al. Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs. USENIX NSDI 2023. usenix.org
9. I. Jang, Z. Yang, Z. Zhang, X. Jin, M. Chowdhury. Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates. SOSP 2023. arXiv:2309.08125