A catalog is a snapshot; ACE is a control loop

Most efficiency tooling hands you a report. It tells you which GPUs are idle, which jobs are over-provisioned, which endpoint is cheapest this week — and then leaves the doing to you. A human works down the list once, and by the time they're at the bottom the top has drifted back out of date. That's a catalog.

ACE is a control loop. It continuously discovers waste, scores the fix against a counterfactual, gates it for safety, and — once it's earned the right — acts. Same brain pointed at every source of waste in the fleet, forever. This post is about what we're building toward, the principles that constrain how we build it, and the concrete sequence that gets us there.

The mission

Merit-order economic dispatch for heterogeneous GPU + API fleets — a marginal-cost-aware dispatch, capacity-commitment, and chargeback layer that sits between applications and in-house GPUs + committed-3P + spot + PAYG APIs, targeting a decomposed, verifiable ~50% compute saving.

Two words in there carry the weight. Decomposed: the savings are broken into attributable levers — utilization reclaim, provider arbitrage, model-routing, caching — not a single opaque number. Verifiable: every dollar is proven against a counterfactual baseline, not asserted. A ~50% headline is only credible if you can show your work, and showing your work is a design constraint, not a marketing afterthought.

The current market landscape

The waste is real and well-measured; what's missing is a system that treats it as one problem.

• Utilization sits near ~50%, structurally. Microsoft's Philly trace study (Jeon et al., ATC '19 [1]) found ~52% average GPU utilization in a production cluster, and the Acme LLM-datacenter characterization (Hu et al., NSDI '24 [2]) found utilization polarized between 0% and 100%. Even flagship training leaves silicon on the table: PaLM (Chowdhery et al., 2022 [3]) set a record at 57.8% Model FLOPs Utilization. The FinOps narrative of 2026 says the same thing in dollars — 30–50% of GPU spend is wasted through over-provisioning.

• Reliability is a first-class cost at scale. Meta's Llama 3 report (2024 [4]) attributes hundreds of unexpected interruptions over a two-month run largely to hardware faults — every one of which is idle, paid-for GPU-time. Efficiency and reliability are not separate budgets.

• Multi-cloud is an economic problem, not a technical one. Sky Computing (Stoica & Shenker, HotOS '21 [5]) argued the barriers to running across providers are mostly economic, and SkyPilot (Yang et al., NSDI '23 [6]) turned cross-cloud spot arbitrage into a broker. Gavel (Narayanan et al., OSDI '20 [7]) made scheduling heterogeneity-aware within a cluster.

Here's the pattern: every existing answer owns one rung of the ladder. API gateways (LiteLLM, Portkey) own request routing but are utilization-blind. In-house packers (Run:ai, KAI) own GPU bin-packing but are third-party-blind. SkyPilot owns provisioning arbitrage but not the request hot-path. Each is a catalog for its own slice. Nobody owns the cross-fleet marginal-cost spill router + commitment sizing + goodput-per-dollar chargeback as one continuous loop. That seam is the whole thesis.

Where ACE is different

ACE is the control loop that spans the rungs. One brain — merit-order economic dispatch — reads every telemetry stream (faults and utilization), prices every destination (owned, reserved, spot, PAYG) on one supply curve, and turns waste into an audited, counterfactually-scored, safety-gated action. The rest of this post is the principles that keep that loop honest, and the road to switching it on.

Six design principles

Every subsystem in ACE is a specific expression of these. Each links to the post that goes deep on it.

1. One brain, swappable backends. In-house GPUs, committed third-party, spot, and pay-as-you-go APIs are four supply curves in four currencies. ACE stacks them into a single marginal-cost merit order and dispatches against it — sunk-cost capacity fills first, PAYG is the peaker of last resort. The substrate is a pluggable adapter behind one uniform interface; the economics are universal. → Merit-order dispatch, One fleet, many clouds.

2. Earn the right to actuate. No action touches a customer cluster until it has proven itself read-only first. Every recommendation climbs a staged trust ladder — record the intended action with no write, then record and notify a human, then finally record, notify, and act on its own. Defaulting to the lowest rung means a misconfiguration can never mutate a fleet. → The invisible tax (SDC), Idle GPUs.

3. Prove every dollar against a counterfactual. ACE never claims a saving in the abstract. It scores every action against "what would the naive baseline have spent on this exact traffic," and for reclaim it attaches a goodput-at-risk receipt to each action — "what served work does removing this capacity endanger?", which is ~0 for a provably-stranded replica. The savings decomposition is designed to resist gaming: you can't book the same GPU-hour twice. → Idle GPUs.

4. Fail safe, always. The most important thing an automated actuator can do is refuse. Storm guards gate every hard eviction on remaining healthy capacity — hold rather than self-starve the fleet below its floor. The compliance filter runs before price optimizes, and if it empties the feasible set ACE raises rather than spills to a non-compliant destination. Safety outranks savings at every branch. → Storm guards.

5. Decide on masked identity. ACE resolves owner → tier and makes its dispatch and reclaim decisions on pseudonymized tokens; the raw workload identity and the salt never leave the customer perimeter. A third-party efficiency layer earns its place past a security review by not ingesting what it doesn't need. → Privacy at the perimeter.

6. Model what you can't afford to break. You cannot crash a 16K-GPU run to test recovery, so the interface to the failure process is the product: ACE's simulator accepts empirical MTBF and checkpoint-latency distributions and validates a policy offline before it's ever pointed at a real fleet. → Modeling the failure process.

If there's a through-line, it's this: measurement and safety are load-bearing, not decorative. Every principle is a mechanism that makes the savings either provable or safe — usually both.

How we get there

The ordering principle for what we build, in what order, is deliberately mechanical: prioritize the biggest headroom, reusing the most existing mechanism, for the least SLA risk. That's why the roadmap looks the way it does — biggest, safest, cheapest-to-build levers first, each one reusing subsystems the last one proved.

Measure first. Coverage — how much of the fleet ACE can see and score — is the headline KPI before anything else, because you cannot reclaim what you don't measure, and you can't get a reclaim program funded without a denominator.

Shadow the biggest, safest lever next. Idle and stranded capacity is provably doing nothing, so its counterfactual goodput-at-risk is ~0 — the system can recommend scaling it to zero with essentially no downside, and prove that out in shadow before ever touching a live cluster.

Prove the loop on already-safe levers. Surface donatable off-peak trough and cache/borrow opportunities as efficiency telemetry, building operator trust with near-zero risk.

Close the loop, canaried. Once shadow observation has proven the counterfactual holds across enough windows, the provably-idle subset moves to fully closed-loop operation — one rack at a time, behind a stage gate, a canary, and the storm guard.

Climb the headroom tiers. Right-sizing and defrag bin-packing come next, then hardware-class migration and proliferation reduction, then the largest dollar lever of all — model-routing migration and quality-gated lower-precision serving — each starting conservatively, because risk rises as the stakes do.

The destination is an autonomous utilization brain. The whole loop wired into one agent: continuous discovery, shadow-scoring, gating, and reclaim, deterministic and auditable end to end.

Adoption follows the same trust ladder the actuator does: a fleet starts with a read-only savings analyzer — feed it a trace, get a decomposed savings report with zero hot-path risk — graduates to shadow mode, then to an inline control plane only once each stage has earned it. The rollout path is the safety path — the loop is never granted more authority than it has already proven it deserves.

The ~50% target is where we're headed, and we mean to get there the way this whole post describes: decomposed into levers you can see, proven against a counterfactual you can check, one trust-earning stage at a time.

---

References

1. M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, F. Yang. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758
2. Q. Hu, Z. Ye, Z. Wang, et al. Characterization of Large Language Model Development in the Datacenter. USENIX NSDI 2024. arXiv:2403.07648
3. A. Chowdhery, S. Narang, J. Devlin, et al. PaLM: Scaling Language Modeling with Pathways. 2022. arXiv:2204.02311
4. Llama Team, AI @ Meta. The Llama 3 Herd of Models. 2024. arXiv:2407.21783
5. I. Stoica, S. Shenker. From Cloud Computing to Sky Computing. HotOS 2021. doi.org
6. Z. Yang, Z. Wu, M. Luo, et al. SkyPilot: An Intercloud Broker for Sky Computing. USENIX NSDI 2023. usenix.org
7. D. Narayanan, K. Santhanam, F. Kazhamiaka, A. Phanishayee, M. Zaharia. Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads. USENIX OSDI 2020. usenix.org