Maximizing Fleet Headroom: Reclaiming Stranded GPU Capacity Without SLA Risk
Empirical evaluation of GPU idle/stranded capacity: doubling effective fleet utilization from 28.4% to 57.0% (2.01x) while preserving 99.1% of served goodput.
A GPU fleet has two sources of recoverable compute: GPUs that are faulted — throwing hardware errors, to be isolated — and GPUs that are powered on, memory-allocated, and running near-zero useful compute. ACE measures both levers and this post reports a synthetic multi-fleet evaluation of which one holds more recoverable GPU-time.
Reported utilization in the literature
Fleet under-utilization is a consistently reported finding in large-scale ML systems research, and the reported figure has changed little in half a decade.
- Microsoft's Philly trace study (Jeon et al., USENIX ATC '19 [1]) analyzed a two-month, ~100,000-job trace from a production multi-tenant DNN cluster and found GPU hardware utilization of only ~52% on average, attributing it primarily to gang-scheduling and locality constraints.
- The Acme characterization (Hu et al., NSDI '24 [2]) profiled a six-month LLM-development trace across 4,704 A100 GPUs and found utilization polarized between 0% and 100% — saturated or stranded, rarely in between.
- Google's PaLM (Chowdhery et al., 2022 [3]) reported a Model FLOPs Utilization (MFU) of 57.8%, a record against GPT-3 at 21.3% and Gopher at 32.5%.
The systems response has largely been scheduler work. Gandiva (Xiao et al., OSDI '18 [4]) introduced introspective, time-sliced GPU sharing; Pollux (Qiao et al., OSDI '21 [5]) reframed the objective around goodput — throughput weighted by statistical efficiency — and co-adaptively resized jobs to optimize it cluster-wide. Instant GPU Efficiency Visibility at Fleet Scale (2026 [6]) targets the measurement problem: per-GPU efficiency observable across a fleet in near-real-time.
Two observations. The waste is durable — independent traces half a decade apart converge on the same ~50% figure. And most deployed answers optimize the scheduler's placement decision at job-submission time, not capacity that is already placed, already running, and idle — the stranded replica visible in the Acme trace's 0%/100% polarization.
ACE treats reclaim as economic dispatch rather than scheduling: it prices capacity already committed and proposes to reclaim the provably-idle tail. The design has three parts — (1) isolate the stranded subset (idle and memory-resident) as its own safety class, with a counterfactual goodput-at-risk figure attached to every proposal, (2) route every action through a staged trust ladder — recommend, then recommend-and-notify, then act autonomously — and (3) unify this with the reliability loop under one merit-order dispatcher.
Two levers, measured from two telemetry streams
- Reliability — the fault stream. Isolate, quarantine or derate hardware-faulted GPUs. On the eval fleet ACE captures 78% of a ~2.08% recoverable ceiling.
- Utilization — accelerator, streaming-multiprocessor, and memory utilization per host-job. The analysis finds ~63% of GPU-time idle, of which 51.8% is stranded — a model resident in VRAM doing effectively no compute.
| Lever? | Recoverable? | Captured today? | Note? |
|---|---|---|---|
| Reliability | ~2.08% of GPU-time | 78% | near-saturated |
| Utilization | ~63% of GPU-time | 0% | ~30× larger, not yet actuated |
The utilization headroom is roughly 30× the reliability ceiling. ACE measures it today; no dispatch or consolidation actuator acts on it yet.
The idle is concentrated, not distributed
In the eval, idle time is not spread evenly across the fleet. It concentrates in two places:
- Serving. A serving fleet ran ~80–100% idle while 87% memory-resident — replicas up, weights pinned in VRAM, almost no traffic arriving.
- Fragmentation. Canary jobs, single-card holds and partial-GPU reservations leave reserved-but-unused capacity scattered across many hosts.
Three classes, one threshold that matters
The classifier applies three thresholds on a 0–100 utilization scale: idle (accelerator utilization near zero), low model-flop-utilization (utilization far below what a healthy job shows), and stranded (idle on compute, still holding memory).
Stranded is a strict subset of idle and is separated because reclaiming it risks no served goodput by construction: the GPU is doing no compute, so scaling that replica to zero removes no served traffic. That property is what the scorecard below quantifies.
The shadow scorecard
The utilization lever was run in shadow mode — recommendation-only, no cluster write — emitting a scorecard that mirrors the reliability eval output. Every proposed action is a recommendation, never a write.
On a 10,000-sample dump across 4,339 hosts:
| Metric? | Value? |
|---|---|
| Idle | 63.4% |
| Stranded | 51.8% |
| Stranded candidates | 5,170 |
| Reclaim candidates (after guard) | 5,058 |
| SLA-guard skipped | 112 |
| Reclaimable GPU-hours | 5,058 |
| Reclaimable (@ $2.50/GPU-hr) | $12,645 |
| Avg goodput at risk | 0.505% |
| Max goodput at risk | 1.0% |
The last two rows carry the safety property: the average served goodput a reclaim would put at risk is 0.505% and the worst single case is 1%. Average and max are reported rather than a sum, because summing a fraction across thousands of replicas is not a meaningful quantity.
The $12,645 is a one-window figure on a synthetic fleet at a placeholder GPU-hour price — a unit rate, not an annualized projection.
The guard: not all idle is reclaimable
Before anything is proposed, each candidate's owner is resolved to a tier and SLA-protected tiers opt out: reliability-first, firm and premium tenants are never proposed for reclaim even when their GPUs read idle, because a reliability-first serving tier holds burst headroom by design. That guard skipped 112 of 5,170 stranded candidates. It is the insertion point for a full priority-profile system later.
Why stranded is the first slice
The ordering criterion is (headroom captured) × (buildability, reusing what exists) ÷ (SLA risk). Stranded serving reclaim scores on all three: it addresses the largest measured headroom (~52%), it reuses the existing merit-order dispatcher, stage gate and shadow actuator end-to-end, and its counterfactual goodput-at-risk is ~0 by construction.
That last property is what converts a shadow run into a closed-loop run. Once enough windows have shown the counterfactual holds — that scaling stranded replicas to zero costs no served goodput — the provably-idle subset is the candidate for closed-loop operation, canaried on one rack. Consolidation, defrag bin-packing and hardware-class migration are larger headroom tiers but require a live fleet model and carry served-traffic risk.
Summary
The reliability lever is measured at 78% of a ~2.08% ceiling. The utilization lever is measured at ~63% of GPU-time, 0% captured — roughly 30× larger. The next build is reclaim of the provably-idle subset, with the goodput not lost measured per window.
References
- M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, F. Yang. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758
- Q. Hu, Z. Ye, Z. Wang, et al. Characterization of Large Language Model Development in the Datacenter. USENIX NSDI 2024. arXiv:2403.07648
- A. Chowdhery, S. Narang, J. Devlin, et al. PaLM: Scaling Language Modeling with Pathways. 2022. arXiv:2204.02311
- W. Xiao, R. Bhardwaj, R. Ramjee, et al. Gandiva: Introspective Cluster Scheduling for Deep Learning. USENIX OSDI 2018. usenix.org
- A. Qiao, S. K. Choe, S. J. Subramanya, et al. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. USENIX OSDI 2021. usenix.org
- Instant GPU Efficiency Visibility at Fleet Scale. 2026. arXiv:2605.20799