Reliability Engineering: Statistical MTBF and ETTR Modeling in GPU Fleets
Distribution-driven failure modeling for large-scale GPU training clusters: predicting MTBF trends, optimizing checkpoint cadences, and automating fault isolation.
For large-scale training, the failure regime that most needs testing is the one that is least affordable to produce.
At 1,024 GPUs a job's mean time to failure is about 7.9 hours [1]. Meta's projection puts a 16,384-GPU job at 1.8 hours and a 131,072-GPU job at 0.23 hours — a failure every fourteen minutes. Runs at that scale are not available as test fixtures for checkpoint cadence, re-placement logic, or preemption budget.
The failure process therefore has to be modeled, and the interface to that model becomes the product: the mechanism is generic and reusable, and the only fleet-specific input is the data.
The current market landscape
Two facts are well established in the literature, and they point in opposite directions.
Failures at scale are frequent, correlated, and dominated by hardware. Revisiting Reliability in Large-Scale ML Research Clusters (Meta, HPCA 2025 [1]) analyzed 11 months across two clusters — 4 million jobs, 150M+ A100 GPU-hours — and found MTTF collapsing roughly two orders of magnitude between an 8-GPU job (47.7 days) and a 1,024-GPU job (7.9 hours). The Llama 3 report (Meta, 2024 [2]) is consistent: over a 54-day pre-training snapshot it recorded 419 unexpected interruptions, ~78% traced to hardware. The Philly trace study (Jeon et al., ATC '19 [3]) and the Acme LLM-development characterization (Hu et al., NSDI '24 [4]) both treat failure as a first-class property of the cluster rather than an exception path.
Recovery is expensive, and its efficiency is a tunable. The classic result is Young's first-order [5] and Daly's higher-order [6] optimal-checkpoint-interval formula — checkpoint too often and write stalls dominate, too rarely and a fail-stop rolls back too much work; the optimum scales as roughly √(2 · δ · MTBF). CheckFreq (Mohan et al., FAST '21 [7]) drove recovery from hours to seconds by tuning checkpoint frequency online within a 3.5% overhead bound. Bamboo (Thorpe et al., NSDI '23 [8]) and Oobleck (Jang et al., SOSP '23 [9]) use redundant pipeline computation and pipeline-template replication to survive failures without a full restart.
The tension: measured failures are non-memoryless and heavy-tailed, while most recovery-policy evaluation assumes a fixed exponential MTBF — a single mean, memoryless, every failure identical to the next. That assumption cannot represent the heavy-tailed regime the traces show.
Where ACE is different
ACE does not own checkpoint mechanics; those stay framework-owned. It owns the economic response to failure — gang re-placement, checkpoint-aware preemption, node quarantine — driven by a failure-process model built on empirical distributions rather than a single average. A fleet's failure history is a direct, swappable input rather than a parameter to guess.
What gets modeled
Two empirical distributions drive the model in place of a single scalar mean-time-to-failure: historical inter-failure timing, and measured per-checkpoint save/load latency. Both are resampled per event, which is what makes the fail-stop process non-memoryless — a bimodal pattern, or a cluster healthy for days that then fails every twenty minutes, produces timing a flat exponential cannot represent.
Every training job carries a fail-stop / checkpoint lifecycle: a failure drawn from the distribution, or absent one a scale-aware estimate calibrated to the ≈7.9-hour-at-1,024-GPU figure from the literature [1], so a large gang fails far more often than a single card. Each fail-stop rolls the job back to its last checkpoint and loses the progress since. The model is observational — it never alters routing, cost, or served goodput — and fully deterministic and replayable, so a recovery policy can be evaluated against the identical failure trace repeatedly.
The metric: ETTR
The headline number is ETTR — Effective Training Time Ratio: productive runtime over wall-clock time, the same construction used to express training goodput as a function of job parameters [1]. It decomposes the unproductive remainder into:
- overhead / init — scheduling barrier, container/driver init, dataset and checkpoint staging;
- wasted progress — compute since the last checkpoint lost to a fail-stop rollback;
- checkpoint-write stall — the synchronous cost of dumping state.
Alongside it: the expected wasted-progress cost implied by the checkpoint cadence (the Young/Daly tradeoff [5][6] made explicit), the scale-aware failure-rate estimate, GPU-hours lost, and a second-order term for the churn a fail-stop induces in neighboring jobs on the same fleet. All of it rolls up into one fleet-level view.
Why the distribution matters
Feeding the model a frequent-failure distribution versus a rare one drives end-to-end ETTR from roughly 0.94 down toward 0.00. A fixed exponential collapses those two fleets into one average and hides the frequently-failing, heavy-tailed regime where recovery-policy differences show up.
Summary
A 16K-GPU training run cannot be broken on purpose to test recovery. The alternative is to make the failure process a first-class, swappable input — sampled from real history, non-memoryless, deterministic to replay — and evaluate recovery policy against it before a live run. The mechanism is shared; each fleet supplies its own failure history.
References
- Meta / AI at Meta. Revisiting Reliability in Large-Scale Machine Learning Research Clusters. IEEE HPCA 2025. arXiv:2410.21680
- Llama Team, AI @ Meta. The Llama 3 Herd of Models. 2024. arXiv:2407.21783
- M. Jeon, S. Venkataraman, A. Phanishayee, et al. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758
- Q. Hu, Z. Ye, Z. Wang, et al. Characterization of Large Language Model Development in the Datacenter. USENIX NSDI 2024. arXiv:2403.07648
- J. W. Young. A First Order Approximation to the Optimum Checkpoint Interval. Communications of the ACM, 1974. (See also Checkpointing à la Young/Daly: An Overview, ACM DOI:10.1145/3549206.3549328)
- J. T. Daly. A Higher Order Estimate of the Optimum Checkpoint Interval for Restart Dumps. Future Generation Computer Systems, 2006. DOI:10.1016/j.future.2004.11.016
- J. Mohan, A. Phanishayee, V. Chidambaram. CheckFreq: Frequent, Fine-Grained DNN Checkpointing. USENIX FAST 2021. usenix.org
- J. Thorpe, P. Zhao, J. Eyolfson, et al. Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs. USENIX NSDI 2023. usenix.org
- I. Jang, Z. Yang, Z. Zhang, X. Jin, M. Chowdhury. Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates. SOSP 2023. arXiv:2309.08125