One fleet, many clouds: cost reduction across a heterogeneous GPU + API estate

Almost nobody runs on one substrate anymore. A serious AI platform team is simultaneously holding owned/on-prem GPUs (capital already sunk), a block of reserved third-party capacity (CoreWeave, Lambda, an AWS Capacity Block), some spot/preemptible headroom that is cheap until it vanishes, and a set of pay-as-you-go model APIs for the overflow it never wants to capacity-plan for. Four substrates, four billing models, four failure modes.

The uncomfortable truth is that most of the cost savings people think they get from multi-cloud never materializes — because each substrate is managed by a different tool, with a different notion of "cheap," and nothing prices them against each other in real time. This post is about treating the whole estate as a single supply curve. (It's the multi-cloud companion to the merit-order dispatch pillar post; read that first if you want the economic framing in full.)

The current market landscape

Two research communities have been circling this problem from opposite ends, and neither quite closes it.

• The scheduling community made clusters heterogeneity-aware. Gavel (Narayanan et al., OSDI '20 [1]) showed that GPUs, TPUs, and accelerators have model-dependent relative performance, and expressed scheduling as an optimization over "effective throughput" — improving objectives like makespan by up to 3.5× versus heterogeneity-agnostic policies. Sia (Subramanya et al., SOSP '23 [2]) extended goodput-optimized scheduling to heterogeneous, elastic clusters. But this line of work optimizes placement within a cluster you already own — it does not price a job against the option of renting a different cloud, or dropping to an API.

• The systems community made the clouds interchangeable. From Cloud Computing to Sky Computing (Stoica & Shenker, HotOS '21 [3]) argued the barriers to multi-cloud are "more economic than technical," and SkyPilot (Yang et al., NSDI '23 [4]) turned that into a broker that runs a job on whichever cloud/region/zone is cheapest and available, exploiting spot-price spreads across providers. This is real arbitrage — but it operates at provisioning granularity (launch a cluster here vs. there), not at per-request dispatch, and it is blind to capacity you have already paid for whose marginal cost is ~0.

• Spot is the cheap tier nobody trusts. Systems like Bamboo (Thorpe et al., NSDI '23 [5]) train resiliently on preemptible instances at a fraction of on-demand price, confirming spot's savings are real — but only for workloads that can absorb preemption. Deciding which traffic may ride spot, and how much SLO risk that buys, remains a policy question left to the operator.

• And the baseline is still ~50% waste. Microsoft's Philly trace study (Jeon et al., ATC '19 [6]) found average GPU utilization of only ~52% in a production cluster. Owning capacity you can't keep full is the most expensive substrate of all — a fixed cost amortized over half the FLOPs.

Put the three threads together and the gap is obvious. Placement schedulers don't see price across substrates; sky brokers don't see the sunk cost of what you already own; spot systems don't decide who is allowed to ride the cheap tier. Nobody is stacking owned + reserved + spot + PAYG into one marginal-cost ordering and dispatching against it request by request.

Where ACE is different

ACE models every destination — an on-prem H100 pool, a reserved CoreWeave block, an AWS spot fleet, an Anthropic/OpenAI API endpoint — as one entry on a single merit-order supply curve, ranked by marginal cost. Sunk-cost tiers price in near zero and fill first; spot sits cheap-but-risky in the middle; PAYG APIs are the flat-high "peaker" of last resort with effectively infinite capacity. The novelty is threefold: (1) the ordering is by true marginal cost, so already-paid capacity is exhausted before a dollar of new spend; (2) a reliability-aware penalty decides which traffic may ride spot without an SLO bet; and (3) a hard compliance gate defines the feasible set before price optimizes within it — a cost optimizer can never spill sensitive traffic into a cloud that isn't allowed to hold it.

The supply stack

New substrates enter through one registry: the place ops points ACE at a new cloud provider's model endpoints and prices, and that provider immediately takes its place on the supply curve alongside everything else already registered. Every destination — in-house, reserved, spot, PAYG — gets priced in the same currency: dollars per unit of usable output. Sunk-cost tiers price in near zero; PAYG prices in high but with effectively unlimited capacity, exactly the peaker-plant role.

The curve isn't flat all the way to its ceiling, either. Push any destination hard enough and cost stops being purely financial — queueing delay starts eating into the latency budget before the dollar cost changes at all. ACE prices that in directly: past a destination's comfortable operating point, its effective cost rises steeply, so the dispatcher naturally spreads load and spills to the next-cheapest substrate before a hot pool tips into latency collapse. Fill from the bottom of the stack up, and you get the behavior a grid operator would recognize: run the nuclear/hydro (owned, sunk) flat-out, bring on the gas peakers (PAYG) only for the last slice of demand, and treat spot as interruptible ballast.

Spot without betting the SLO

Spot is where naive cost optimizers get burned: it's the cheapest GPU-hour on the board, so a pure-price policy happily lands latency-critical serving on it — and then a preemption takes the SLO with it. ACE handles this with an explicit, finite penalty rather than a blanket ban:

• In pure-price cost mode, latency-critical traffic may land on cheap spot exactly as marginal cost dictates.
• In reliability mode, stable tiers outrank spot for latency-critical traffic — because a spot reclaim risks an SLO violation — but the penalty stays finite, so spot remains reachable as a last-resort fallback before PAYG. Interruptible and offline traffic is never penalized: spot is its ideal ballast.

A separate provider-preference weighting is the knob for everything price can't express: egress-cost asymmetries, a provider you're trying to drain, a reliability reputation, a committed-spend floor you want to burn down. It nudges the merit order without overriding it.

The gate that outranks price

Here is the design decision that separates a sellable multi-cloud optimizer from a liability. Before ACE ranks anything by cost, it filters to the destinations a given workload is actually allowed to run on — data-residency, tenant isolation, provider allow-lists. Price optimizes only within that feasible set. And it fails safe:

If the compliance filter empties the set, ACE raises rather than spills to a non-compliant destination. Never spill to a non-compliant dest.

This is the multi-cloud analog of security-constrained economic dispatch in power grids: the constraints define the feasible region first, and the price optimization runs inside it — never the other way around. A cheaper GPU in the wrong jurisdiction is not a cheaper GPU; it's an incident.

Reliability is cost

The last piece is that spill isn't only a pricing decision — it's how ACE converts a reliability event into a cost event and back. When a destination starts failing, ACE trips it out of the feasible set and dispatch re-routes to the next-cheapest compliant option; a rate-limit response shrinks a destination's effective capacity, a server error drives its marginal cost to infinity. The same merit-order machinery that saves money on a calm day reroutes around a provider outage on a bad one — because in this model an unreachable GPU and an infinitely expensive GPU are the same thing.

The takeaway

Multi-cloud cost reduction is not a procurement problem you solve once at contract time; it's a dispatch problem you solve continuously. The savings come from three moves that no single existing tool makes together: exhaust the capacity you've already paid for before spending a new dollar, let interruptible work — and only interruptible work — ride spot, and refuse on principle to let a cost optimizer cross a compliance boundary. Stack the substrates into one merit order, gate it on feasibility, and the cheapest safe FLOP wins every request.

---

References

1. D. Narayanan, K. Santhanam, F. Kazhamiaka, A. Phanishayee, M. Zaharia. Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads. USENIX OSDI 2020. usenix.org
2. S. J. Subramanya, D. Arfeen, S. Lin, A. Qiao, Z. Jia, G. R. Ganger. Sia: Heterogeneity-aware, Goodput-optimized ML-cluster Scheduling. ACM SOSP 2023. doi.org
3. I. Stoica, S. Shenker. From Cloud Computing to Sky Computing. HotOS 2021. doi.org
4. Z. Yang, Z. Wu, M. Luo, et al. SkyPilot: An Intercloud Broker for Sky Computing. USENIX NSDI 2023. usenix.org
5. J. Thorpe, P. Zhao, J. Eyolfson, et al. Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs. USENIX NSDI 2023. usenix.org
6. M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, F. Yang. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758