The order of magnitude is in the idle GPUs, not the broken ones

If you run a GPU fleet, you have two ways to get compute back. You can chase the GPUs that are broken — faulted, poisoned, throwing hardware errors — and isolate them fast. Or you can chase the GPUs that are doing nothing — powered on, memory allocated, and burning ~zero useful compute.

Most reliability tooling is built for the first. We built ACE — think merit-order economic dispatch for GPUs, with a reliability brain on top — to do both, and then we ran a synthetic multi-fleet evaluation to find out which lever actually holds the money.

The answer was lopsided enough to reorder the roadmap.

The current market landscape

Fleet under-utilization is not a secret — it is one of the most consistently reported findings in large-scale ML systems research, and the number has barely moved in half a decade.

• Microsoft's Philly trace study (Jeon et al., USENIX ATC '19 [1]) analyzed a two-month, ~100,000-job trace from a production multi-tenant DNN cluster and found GPU hardware utilization of only ~52% on average, with gang-scheduling and locality constraints as primary culprits.
• The Acme characterization (Hu et al., NSDI '24 [2]) profiled a six-month LLM-development trace across 4,704 A100 GPUs and found utilization is polarized between 0% and 100% — the fleet is either saturated or stranded, rarely in between.
• The ceiling is visible even in flagship runs. Google's PaLM (Chowdhery et al., 2022 [3]) reported a Model FLOPs Utilization (MFU) of 57.8% — and that was a record that threw earlier models into relief: GPT-3 at 21.3%, Gopher at 32.5%. Even the best-tuned training pipelines leave nearly half the silicon on the table.

The systems community's answer has largely been better schedulers. Gandiva (Xiao et al., OSDI '18 [4]) introduced introspective, time-sliced GPU sharing; Pollux (Qiao et al., OSDI '21 [5]) reframed the objective around goodput — throughput weighted by statistical efficiency — and co-adaptively resized jobs to optimize it cluster-wide. More recent work such as Instant GPU Efficiency Visibility at Fleet Scale (2026 [6]) targets the measurement problem directly: making per-GPU efficiency observable across an entire fleet in near-real-time.

Two things stand out. First, the waste is structural and durable — independent traces half a decade apart converge on the same ~50% figure, so this is not a transient tuning problem. Second, almost every deployed answer optimizes the scheduler's placement decision at job-submission time. Far less attention goes to the capacity that is already placed, already running, and quietly doing nothing — the stranded replica that shows up so vividly in the Acme trace's 0%/100% polarization.

Where ACE is headed

ACE treats reclaim as economic dispatch, not scheduling: instead of packing the next job better, it continuously prices the capacity already committed and proposes to reclaim the provably-idle tail. The vision has three parts — (1) isolate the stranded subset (idle and memory-resident) as its own safety class, with a counterfactual goodput-at-risk receipt attached to every proposal, (2) route every action through a staged trust ladder — recommend first, then recommend-and-notify, then finally act autonomously — so a recommendation earns the right to actuate, and (3) unify this with the reliability loop under one merit-order brain. The rest of this post is what that looks like against a real evaluation.

Two levers, measured from two streams

ACE reads two independent telemetry streams and turns each into a headroom number:

• Reliability — the fault stream. Fast isolate / quarantine / derate of hardware-faulted GPUs. On the eval fleet ACE captures 78% of a ~2.08% recoverable ceiling. That lever is near-saturated: there isn't much left to win, and we're already winning most of it.

• Utilization — accelerator, streaming-multiprocessor, and memory utilization per host-job. Here the analysis finds ~63% of GPU-time idle, of which 51.8% is stranded — a model resident in VRAM doing effectively no compute.

Put those side by side and the shape of the problem is obvious:

The utilization headroom is roughly 30× the entire reliability ceiling. ACE measures it today; wiring a dispatch and consolidation actuator to act on it is the next build — and it's where the order of magnitude lives.

Idle is structural, not random

The temptation is to read "63% idle" as noise — diurnal troughs, warmup, jitter averaging out. It isn't. In the eval the idle concentrates, and it concentrates in the two places you'd least want it:

• Serving. A serving fleet ran ~80–100% idle while 87% memory-resident — replicas held up, model weights pinned in VRAM, and almost no traffic hitting them.
• Fragmentation. Canary jobs, single-card holds, partial-GPU reservations — reserved-but-unused capacity scattered across many boxes.

That's not randomness you can average away. It's capacity that is structurally parked, and structural waste is the kind you can actually schedule against.

Three classes, one threshold that matters

The classifier at the heart of this is deliberately simple: three thresholds on a 0–100 utilization scale define idle (accelerator utilization near zero), low model-flop-utilization (utilization far below what a healthy job should show), and stranded (idle on compute, yet still holding memory).

The class that matters is stranded: idle and memory-resident. A model pinned in VRAM serving ~zero traffic. It's a strict subset of idle, but it earns its own label for one reason — reclaiming it risks no served goodput. The GPU is already doing nothing; scaling that replica to zero takes nothing away from anyone.

That's the wedge. You don't have to win the argument about burst headroom or "what if traffic spikes" to reclaim a stranded replica. By construction it is serving nothing.

The shadow scorecard

We ran the utilization lever the same way we run the reliability loop — in shadow mode, recommendation-only, no cluster write — and emitted a scorecard that mirrors the reliability eval output. Every proposed action is provably a recommendation, never a write.

On a 10,000-sample dump across 4,339 hosts:

Read the last two rows carefully, because they're the whole safety argument. The average served goodput a reclaim would put at risk is half a percent, and the worst single case is 1% — that's what "stranded" guarantees. We report avg and max rather than a sum, because summing a fraction across thousands of replicas is a number designed to mislead.

That $12,645 is a one-window figure on a synthetic fleet at a placeholder GPU-hour price — treat it as a unit rate, not an annualized promise. The point isn't the dollar amount; it's that it comes with a 0.5% goodput-at-risk receipt attached.

The guard: not all idle is reclaimable

The 112 "guard skipped" candidates matter as much as the 5,058 reclaimed ones. Before anything is proposed, each candidate's owner is resolved to a tier, and SLA-protected tiers opt out — reliability-first, firm, and premium tenants are never proposed for reclaim, even when their GPUs look idle. A reliability- first serving tier legitimately holds burst headroom; that capacity is supposed to sit ready. The guard is the seam where a full priority-profile system plugs in later, but even the first slice refuses to treat all idle as fungible.

Why stranded is the first slice

The roadmap ordering principle is: maximize (headroom captured) × (buildability, reusing what exists) ÷ (SLA risk). Stranded serving reclaim wins on all three:

• it attacks the largest measured headroom (~52%),
• it reuses the existing merit-order brain, stage gate, and shadow actuator end-to-end, and
• it carries the lowest SLA risk, because the counterfactual goodput-at-risk is ~0 by construction.

That last property is what lets the shadow run become a closed-loop run. Once enough windows have proven the counterfactual holds — that scaling stranded replicas to zero costs no served goodput — that's the evidence to flip just the provably-idle subset to closed-loop, canaried on one rack. Consolidation, defrag bin-packing, and hardware-class migration are bigger headroom tiers, but they need a live fleet model and carry real risk. Stranded reclaim doesn't. It's the safe place to earn the right to actuate.

The takeaway

We spent a long time getting good at the reliability lever, and we are good at it — 78% of a 2% ceiling. But the eval was blunt: the reliability lever is a rounding error next to the idle one. The GPUs that are quietly doing nothing cost 30× more than the ones that are broken.

The unglamorous first move is to reclaim the replicas that are provably doing nothing, measure the goodput you didn't cost anyone, and let that receipt buy you the right to close the loop. That's the next build.

---

References

1. M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, F. Yang. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758
2. Q. Hu, Z. Ye, Z. Wang, et al. Characterization of Large Language Model Development in the Datacenter. USENIX NSDI 2024. arXiv:2403.07648
3. A. Chowdhery, S. Narang, J. Devlin, et al. PaLM: Scaling Language Modeling with Pathways. 2022. arXiv:2204.02311
4. W. Xiao, R. Bhardwaj, R. Ramjee, et al. Gandiva: Introspective Cluster Scheduling for Deep Learning. USENIX OSDI 2018. usenix.org
5. A. Qiao, S. K. Choe, S. J. Subramanya, et al. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. USENIX OSDI 2021. usenix.org
6. Instant GPU Efficiency Visibility at Fleet Scale. 2026. arXiv:2605.20799