ACE Engineering Principles: Autonomous Compute Efficiency for Foundation Models
The core mission and architectural principles behind ACE: continuous closed-loop dispatch, merit-order capacity routing, and verifiable 50%+ compute savings across AI infrastructure.
Most efficiency tooling produces a report: which GPUs are idle, which jobs are over-provisioned, which endpoint is cheapest this week. The remediation is left to an operator, and the findings age while that operator works down the list. That is a catalog.
ACE is a control loop: it discovers waste continuously, scores each fix against a counterfactual, gates it for safety, and — once the stage gate allows it — acts. This post covers the mission, the principles that constrain the implementation, and the sequence of work between the two.
The mission
Merit-order economic dispatch for heterogeneous GPU + API fleets — a marginal-cost-aware dispatch, capacity-commitment, and chargeback layer that sits between applications and in-house GPUs + committed-3P + spot + PAYG APIs, targeting a decomposed, verifiable ~50% compute saving.
Two terms are load-bearing. Decomposed: savings are broken into attributable levers — utilization reclaim, provider arbitrage, model-routing, caching — rather than reported as one aggregate. Verifiable: each dollar is scored against a counterfactual baseline rather than asserted.
The current market landscape
Utilization sits near ~50%, structurally. Microsoft's Philly trace study (Jeon et al., ATC '19 [1]) found ~52% average GPU utilization in a production cluster, and the Acme LLM-datacenter characterization (Hu et al., NSDI '24 [2]) found utilization polarized between 0% and 100%. PaLM (Chowdhery et al., 2022 [3]) set a record at 57.8% Model FLOPs Utilization. FinOps reporting in 2026 puts 30–50% of GPU spend as wasted through over-provisioning.
Reliability is a first-class cost at scale. Meta's Llama 3 report (2024 [4]) attributes hundreds of unexpected interruptions over a two-month run largely to hardware faults, each of which is idle, paid-for GPU-time.
Multi-cloud is an economic problem, not a technical one. Sky Computing (Stoica & Shenker, HotOS '21 [5]) argued the barriers to running across providers are mostly economic, and SkyPilot (Yang et al., NSDI '23 [6]) implemented cross-cloud spot arbitrage as a broker. Gavel (Narayanan et al., OSDI '20 [7]) made scheduling heterogeneity-aware within a cluster.
Existing systems each cover one rung. API gateways (LiteLLM, Portkey) own request routing but not utilization. In-house packers (Run:ai, KAI) own GPU bin-packing but not third-party capacity. SkyPilot owns provisioning arbitrage but not the request hot-path. The cross-fleet marginal-cost spill router + commitment sizing + goodput-per-dollar chargeback is not covered as one continuous loop.
Where ACE is different
ACE spans those rungs with one dispatch layer: it reads every telemetry stream (faults and utilization), prices every destination (owned, reserved, spot, PAYG) on one supply curve, and converts waste into an audited, counterfactually-scored, safety-gated action.
Six design principles
1. One brain, swappable backends. In-house GPUs, committed third-party, spot, and pay-as-you-go APIs are four supply curves in four currencies. ACE stacks them into a single marginal-cost merit order and dispatches against it — sunk-cost capacity fills first, PAYG is the peaker of last resort. The substrate is a pluggable adapter behind one uniform interface. → Merit-order dispatch, One fleet, many clouds.
2. Earn the right to actuate. No action touches a customer cluster until it has run read-only first. Every recommendation climbs a staged trust ladder: record the intended action with no write, then record and notify a human, then record, notify, and act. The default is the lowest rung, so a misconfiguration cannot mutate a fleet. → Silent data corruption (SDC), Idle versus faulted GPU-time.
3. Prove every dollar against a counterfactual. Each action is scored against what the naive baseline would have spent on the same traffic, and reclaim actions carry a goodput-at-risk receipt — what served work removing this capacity endangers, which is ~0 for a provably-stranded replica. The decomposition prevents booking the same GPU-hour twice. → Idle versus faulted GPU-time.
4. Fail safe, always. Storm guards gate every hard eviction on remaining healthy capacity: the loop holds rather than taking the fleet below its floor. The compliance filter runs before price optimizes, and if it empties the feasible set ACE raises rather than spilling to a non-compliant destination. → Storm guards.
5. Decide on masked identity. ACE resolves owner → tier and makes dispatch and reclaim decisions on pseudonymized tokens; the raw workload identity and the salt never leave the customer perimeter. → Pseudonymizing GPU telemetry.
6. Model what you can't afford to break. A 16K-GPU run cannot be crashed to test recovery, so the interface to the failure process is itself the product: ACE's simulator accepts empirical MTBF and checkpoint-latency distributions and validates a policy offline before it is pointed at a real fleet. → Simulating fail-stop failure.
Each principle is a mechanism that makes a saving either provable or safe.
How we get there
The ordering rule is mechanical: largest headroom, reusing the most existing mechanism, for the least SLA risk.
Measure first. Coverage — how much of the fleet ACE can see and score — is the headline KPI before anything else, because reclaim requires a denominator.
Shadow the biggest, safest lever next. Idle and stranded capacity is provably doing nothing, so its counterfactual goodput-at-risk is ~0; scaling it to zero can be recommended and validated in shadow before touching a live cluster.
Prove the loop on already-safe levers. Donatable off-peak trough and cache/borrow opportunities surface as efficiency telemetry, at near-zero risk.
Close the loop, canaried. Once shadow observation has held across enough windows, the provably-idle subset moves to closed-loop operation one rack at a time, behind a stage gate, a canary, and the storm guard.
Climb the headroom tiers. Right-sizing and defrag bin-packing, then hardware-class migration and proliferation reduction, then the largest dollar lever — model-routing migration and quality-gated lower-precision serving.
The destination is an autonomous utilization brain: continuous discovery, shadow-scoring, gating, and reclaim in one agent, deterministic and auditable end to end.
Adoption follows the same ladder as the actuator. A fleet starts with a read-only savings analyzer — feed it a trace, get a decomposed savings report with no hot-path risk — then shadow mode, then an inline control plane, each stage gated on the last.
References
- M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, F. Yang. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758
- Q. Hu, Z. Ye, Z. Wang, et al. Characterization of Large Language Model Development in the Datacenter. USENIX NSDI 2024. arXiv:2403.07648
- A. Chowdhery, S. Narang, J. Devlin, et al. PaLM: Scaling Language Modeling with Pathways. 2022. arXiv:2204.02311
- Llama Team, AI @ Meta. The Llama 3 Herd of Models. 2024. arXiv:2407.21783
- I. Stoica, S. Shenker. From Cloud Computing to Sky Computing. HotOS 2021. doi.org
- Z. Yang, Z. Wu, M. Luo, et al. SkyPilot: An Intercloud Broker for Sky Computing. USENIX NSDI 2023. usenix.org
- D. Narayanan, K. Santhanam, F. Kazhamiaka, A. Phanishayee, M. Zaharia. Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads. USENIX OSDI 2020. usenix.org