Heterogeneous Accelerator Management: Multi-Cloud GPU Fleet Optimization
Unifying owned GPUs, reserved capacity, spot instances, and API providers into a single merit-order dispatch engine for maximum goodput and cost efficiency.
Most AI platform teams run on several substrates at once: owned/on-prem GPUs (capital already sunk), a block of reserved third-party capacity (CoreWeave, Lambda, an AWS Capacity Block), some spot/preemptible headroom that is cheap until it is reclaimed, and pay-as-you-go model APIs for overflow. Four substrates, four billing models, four failure modes — each typically managed by a different tool with a different notion of "cheap," and nothing pricing them against each other in real time.
This post covers treating the whole estate as a single supply curve. It is the multi-cloud companion to the merit-order dispatch post, which has the economic framing in full.
The current market landscape
Two research communities approach this from opposite ends.
The scheduling community made clusters heterogeneity-aware. Gavel (Narayanan et al., OSDI '20 [1]) showed that GPUs, TPUs, and accelerators have model-dependent relative performance, and expressed scheduling as an optimization over "effective throughput" — improving objectives like makespan by up to 3.5× versus heterogeneity-agnostic policies. Sia (Subramanya et al., SOSP '23 [2]) extended goodput-optimized scheduling to heterogeneous, elastic clusters. This line of work optimizes placement within a cluster you already own; it does not price a job against renting a different cloud or dropping to an API.
The systems community made the clouds interchangeable. From Cloud Computing to Sky Computing (Stoica & Shenker, HotOS '21 [3]) argued the barriers to multi-cloud are "more economic than technical," and SkyPilot (Yang et al., NSDI '23 [4]) turned that into a broker that runs a job on whichever cloud/region/zone is cheapest and available, exploiting spot-price spreads across providers. It operates at provisioning granularity — launch a cluster here vs. there — not per-request dispatch, and it does not model capacity already paid for whose marginal cost is ~0.
Spot savings are real but conditional. Systems like Bamboo (Thorpe et al., NSDI '23 [5]) train resiliently on preemptible instances at a fraction of on-demand price, confirming spot's savings for workloads that can absorb preemption. Which traffic may ride spot, and the SLO risk that carries, is left as a policy question for the operator.
The measured baseline is ~50% waste. Microsoft's Philly trace study (Jeon et al., ATC '19 [6]) found average GPU utilization of only ~52% in a production cluster — a fixed cost amortized over half the FLOPs.
The gap across the three threads: placement schedulers do not see price across substrates; sky brokers do not see the sunk cost of owned capacity; spot systems do not decide which traffic may ride the cheap tier. None stacks owned + reserved + spot + PAYG into one marginal-cost ordering and dispatches against it request by request.
Where ACE is different
ACE models every destination — an on-prem H100 pool, a reserved CoreWeave block, an AWS spot fleet, an Anthropic/OpenAI API endpoint — as one entry on a single merit-order supply curve ranked by marginal cost. Three properties: the ordering is by true marginal cost, so already-paid capacity is exhausted before new spend; a reliability-aware penalty decides which traffic may ride spot; and a hard compliance gate defines the feasible set before price optimizes within it.
The supply stack
New substrates enter through one registry: ops points ACE at a new provider's model endpoints and prices, and that provider takes its place on the supply curve alongside everything already registered. Every destination is priced in the same currency — dollars per unit of usable output. Sunk-cost tiers price near zero and fill first; spot sits cheap-but-interruptible in the middle; PAYG APIs are the flat-high peaker of last resort with effectively infinite capacity.
The curve is not flat to its ceiling. Push any destination hard enough and queueing delay starts consuming the latency budget before dollar cost changes at all. ACE prices that in: past a destination's comfortable operating point its effective cost rises steeply, so the dispatcher spreads load and spills to the next-cheapest substrate before a hot pool reaches latency collapse. Filling from the bottom up reproduces grid behavior: owned/sunk capacity runs flat-out, PAYG covers the last slice of demand, spot serves as interruptible ballast.
Spot without betting the SLO
Spot is the cheapest GPU-hour on the board, so a pure-price policy lands latency-critical serving on it, where a preemption takes the SLO with it. ACE applies an explicit, finite penalty rather than a blanket ban:
- In pure-price cost mode, latency-critical traffic may land on cheap spot exactly as marginal cost dictates.
- In reliability mode, stable tiers outrank spot for latency-critical traffic, since a spot reclaim risks an SLO violation — but the penalty stays finite, so spot remains reachable as a last-resort fallback before PAYG. Interruptible and offline traffic is never penalized.
A separate provider-preference weighting covers what price cannot express: egress-cost asymmetries, a provider being drained, a reliability reputation, a committed-spend floor. It shifts the merit order without overriding it.
The gate that outranks price
Before ACE ranks anything by cost, it filters to the destinations a workload is allowed to run on — data-residency, tenant isolation, provider allow-lists. Price optimizes only within that feasible set, and the filter fails safe:
If the compliance filter empties the set, ACE raises rather than spills to a non-compliant destination. Never spill to a non-compliant dest.
This is the multi-cloud analog of security-constrained economic dispatch in power grids: constraints define the feasible region first, and price optimization runs inside it.
Reliability is cost
Spill is also how a reliability event becomes a cost event. When a destination starts failing, ACE trips it out of the feasible set and dispatch re-routes to the next-cheapest compliant option; a rate-limit response shrinks a destination's effective capacity, and a server error drives its marginal cost to infinity. The same merit-order machinery that lowers cost on a calm day reroutes around a provider outage, because in this model an unreachable GPU and an infinitely expensive GPU are identical.
Summary
Multi-cloud cost reduction is a continuous dispatch problem rather than a one-time procurement decision. Three moves combine here: exhaust already-paid capacity before new spend, restrict spot to interruptible work under a finite reliability penalty, and keep the compliance boundary above the cost optimizer. Stack the substrates into one merit order, gate it on feasibility, and the cheapest feasible destination wins each request.
References
- D. Narayanan, K. Santhanam, F. Kazhamiaka, A. Phanishayee, M. Zaharia. Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads. USENIX OSDI 2020. usenix.org
- S. J. Subramanya, D. Arfeen, S. Lin, A. Qiao, Z. Jia, G. R. Ganger. Sia: Heterogeneity-aware, Goodput-optimized ML-cluster Scheduling. ACM SOSP 2023. doi.org
- I. Stoica, S. Shenker. From Cloud Computing to Sky Computing. HotOS 2021. doi.org
- Z. Yang, Z. Wu, M. Luo, et al. SkyPilot: An Intercloud Broker for Sky Computing. USENIX NSDI 2023. usenix.org
- J. Thorpe, P. Zhao, J. Eyolfson, et al. Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs. USENIX NSDI 2023. usenix.org
- M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, F. Yang. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758