Merit-order dispatch for GPUs: what the power grid knows that schedulers don't

A power grid operator dispatching 40 GW of load does not reason about individual power plants. They build one supply curve — every generator stacked left to right by marginal cost, cheapest first — and fill it from the bottom until supply meets demand. Nuclear and hydro (sunk cost, ~zero marginal) run flat-out; combined-cycle gas fills the middle; and the expensive peaker plants only light up for the last slice of demand on the hottest afternoon. This is economic dispatch, and it has been the operating discipline of electricity markets for the better part of a century [1].

An AI shop running in-house GPUs, reserved third-party capacity (PTU/GSU), spot, and pay-as-you-go APIs has exactly this problem — a heterogeneous stack of supply with marginal costs that differ by one to two orders of magnitude — and almost nobody dispatches it that way. The cheapest capacity (owned GPUs you've already paid for) sits half-idle while the most expensive rung (PAYG) absorbs forecastable baseline load. That is the equivalent of running your gas peakers 24/7 while the paid-off nuclear plant idles. No grid operator would survive it.

ACE is, in one line, merit-order economic dispatch for heterogeneous GPU + API fleets. This post is the mental model, and the mechanism underneath it.

The current market landscape

The two disciplines this problem sits between — power-systems dispatch and ML cluster scheduling — are each mature. What's missing is the bridge.

Economic dispatch is a solved problem in the domain that invented it. The merit-order formulation — rank generators by ascending marginal cost, dispatch cheapest-first to meet demand at least total cost — is textbook power systems, and the modern literature extends it to multi-area grids, uncertain (renewable) supply, and reserve co-optimization [1]. Crucially, real grids do not run pure merit order: they run security-constrained economic dispatch, where physical constraints (transmission limits, contingency reserves) define the feasible set first, and cost optimizes only within it. Hold onto that idea — it's the part most GPU routers get wrong.

ML cluster scheduling is also mature — but it optimizes a different variable. The influential systems here schedule placement: which job lands on which GPU, when. Gandiva [2] introduced introspective, time-sliced GPU sharing and job migration. Pollux [3] reframed the objective around goodput — throughput weighted by statistical efficiency — and co-adaptively resized jobs to maximize it cluster-wide. Sia [4] extended goodput-optimized scheduling to heterogeneous clusters, matching elastic jobs to the right GPU type and count. This is a rich, deployed body of work — but it answers "where should this job run inside my owned cluster?", not "given live prices across four different substrates, where is the marginal token cheapest to serve right now, and how much cheap capacity should I have bought last week?"

Provisioning arbitrage is a third, separate rung. SkyPilot [5] treats the multi-cloud market as a two-sided broker, launching replicas on the cheapest available spot/on-demand capacity across providers and regions. It's excellent at where to launch a replica — but it operates at provisioning time, not on the request hot path, and it has no view of your in-house fleet's utilization or your committed-3P blocks.

And the waste this leaves on the table is not hypothetical. Microsoft's two-month Philly trace study found average GPU utilization of only ~52% on a production multi-tenant cluster [6] — a number that has barely moved across the traces published since. Every rung of the stack is individually well-optimized; the fleet as a whole is not.

The products map onto the same fragmentation. LiteLLM and the API-gateway tier own provider normalization and fallback chains — but they are marginal-cost- blind: they weight traffic across endpoints with no model of where each destination sits on its cost-vs-utilization curve, and no concept of an in-house fleet at all (3P-only). Run:ai / KAI own in-house Kubernetes GPU packing — but they're 3P-blind: they optimize the supply of GPUs you own and have no idea a PTU block or a PAYG endpoint exists, so they can't make the buy-vs-spill decision. Each owns one rung. Nobody runs the whole ladder as a single merit order.

Where ACE is different

ACE's thesis is that a control loop is economic dispatch — continuously discover the supply curve, dispatch cheapest-first, and safety-gate the result — applied across the entire heterogeneous fleet at once. Concretely, three things the incumbents structurally lack:

1. One supply curve across every substrate. In-house, reserved-3P, spot, on-demand, and PAYG all register as destinations behind a single uniform interface, and one marginal-cost merit order arbitrates across all of them. "One brain, swappable backends." A 3P-only version of this is just a gateway; an in-house-only version is just a packer. Unifying them is what turns scheduling into economic dispatch.
2. Security-constrained dispatch, not pure price. Capability and compliance constraints define the feasible set before cost optimizes within it — cost can never override a data-residency rule, exactly as a grid's transmission limits gate its dispatch.
3. The slow loop and the ledger. Merit order is the fast loop. The savings are mostly decided by the slow loop — sizing committed capacity to forecast demand so you never pay peaker rates for baseline load — and proven by a goodput-per-dollar chargeback ledger that measures the counterfactual.

The rest of this post walks the mechanism.

The supply curve: cost curves and the stutter knee

Every destination carries a cost curve, and its shape encodes the physics of its tier. An in-house GPU is baseload: you've already paid for it, so its marginal cost to the capacity ceiling is ~zero — and then it's a hard cliff (you cannot serve past what you own). Reserved third-party capacity behaves the same way up to its committed cap. Spot is cheap and interruptible, with a short eviction warning before it disappears. PAYG is the gas peaker: expensive per token, but effectively infinite and instantly available, which is exactly why you want it for the unpredictable tail and nowhere else.

The subtlety the grid analogy makes obvious is the stutter knee. A generator's marginal cost isn't flat right up to its nameplate rating — push it toward the limit and you pay in heat rate and ramp stress. A GPU serving latency-sensitive traffic is the same: past some utilization point, queueing delay starts eating your latency SLO, so the effective marginal cost rises even though the dollar cost hasn't. ACE prices that in directly, and the dispatch rule that falls out is precise: fill each tier to its stutter knee, not to its cap. A naive packer that fills to 100% looks efficient on a utilization dashboard and quietly violates every latency SLO it touches.

Dispatch: rank, but constrain first

The fast loop is two steps, in this order — and the order is the whole ballgame.

Step 1 — feasibility (the security constraint). ACE filters the fleet down to the set that can legally and physically serve this request: capability mask (does it have the model, the VRAM class, the accelerator family, the interconnect for a tensor-parallel shard?) and — critically — the compliance gate, which runs before merit order and which cost and latency can never override. An EU-personal-data request is simply not eligible for a US-jurisdiction destination, at any price. This is security-constrained economic dispatch, lifted straight from the grid: constraints define the feasible set, price optimizes inside it. A router that lets a cheap destination win a request it was never allowed to serve isn't cheap — it's a compliance incident.

Step 2 — merit order. Among the survivors, ACE ranks by effective marginal cost plus two GPU-specific correction terms and picks the cheapest: a warmth term (a model already resident in VRAM is cheaper to serve than one that needs a cold load — the residency is a form of already-paid-for readiness), and the latency penalty from the stutter knee above. One honest-accounting detail worth calling out: ranking biases like a provider-preference weight adjust where a destination sorts, but never mutate the underlying cost curve — the ledger always re-derives the true marginal, so a routing preference can never quietly inflate the savings number.

Spill: reliability is efficiency

Here's where merit order and reliability turn out to be the same mechanism.

When a destination fails — a server error, a timeout, a connection refused — ACE trips it out of the feasible set, which means the next request automatically re-dispatches to the next-cheapest healthy destination. There is no separate failover subsystem; spill-to-next-cheapest is just merit order running over a fleet with one fewer generator online. A grid operator losing a unit doesn't invoke a different control loop — the supply curve just shifts and the next-cheapest MW fills in.

The pricing view makes the unification exact. A rate-limit response from a destination means its usable capacity just shrank — the curve's cap moves left. A server error means its effective marginal cost just went to infinity — it drops off the right edge of the merit order entirely. Reliability signals are cost-curve updates. A degrading-but-alive link doesn't get cut; it gets priced down so it sheds load gracefully rather than failing hard — the derate, not the trip.

Early results on this derate-instead-of-failover behavior are strong: pricing a throttling node down and continuing to serve from it, instead of evacuating to a pricier tier, meaningfully improves goodput-per-dollar on degraded-node scenarios — exactly the kind of win that compounds across a large, noisy fleet.

The slow loop: unit commitment is where the money is

Everything above is the fast loop — dispatch given the capacity you have. But a grid operator makes the expensive decision hours ahead, in unit commitment: deciding which units to bring online before demand materializes, so the cheap baseload is already running when the morning ramp hits and you never fire a peaker for load you could have seen coming.

The compute analog is commitment sizing, and it's where most of the savings actually live. ACE's slow loop forecasts demand and sizes committed/in-house capacity to roughly the p90 of the curve — enough that PAYG only ever absorbs the genuine tail, never the forecastable baseline. The operating principle is a single sentence: never pay peaker rates for baseload demand. Around it sit two more grid-shaped levers: valley-filling (push deferrable offline work — batch indexing, evals, training — into the diurnal troughs online traffic leaves empty, the way grids charge storage overnight), and spot arbitrage (a live price feed chases the cheapest interruptible capacity across clouds, with a proactive drain on the eviction notice so a reclaim is a graceful checkpoint, not a failure).

Directionally, this is where the largest share of total savings lives: taking reserved-capacity utilization from a fraction of the fleet up toward its full committed level, and doing it by forecasting demand rather than reacting to it after the fact.

The ledger: a saving you can't measure isn't a saving

A grid settles every interval against a market price so everyone can see what dispatch actually cost. ACE does the same with a goodput-per-marginal-dollar ledger built from one per-request event spine. Two properties make it more than a dashboard:

• It's counterfactual. Every request records what a naive baseline would have cost on that exact traffic — the "what if we'd just sent everything to PAYG?" line. Savings are decomposed four ways — utilization, arbitrage, model-routing, caching — so you can see which lever earned the money, not just a blended percentage.
• It's anti-gamed. A saving only counts if served-fraction, SLO-attainment, and bounded cache false-hit rate all held. Cheaper-but-dropped or cheaper-but-degraded does not score. You cannot book savings by quietly serving worse — the one failure mode every "we cut costs 50%" pitch hides.

This ledger is also what makes the savings auditable after the fact: because every dispatch decision and its counterfactual land on one event spine, any claimed number can be traced back to the exact requests that produced it.

Why one brain, and not four tools

You could assemble a facsimile of this from parts: a gateway for provider fallback, a K8s packer for in-house, a provisioner for spot, a router for model choice. Teams do. But each tool optimizes its own rung blind to the others — the gateway weights traffic with no idea the in-house pool is past its knee; the provisioner launches a spot replica for load PAYG was already covering; the model router downgrades a request on a box that was idle anyway. The grid lesson is that economic dispatch is a global optimization over one supply curve — decompose it into four independent local optimizers and you get four locally-sensible decisions that are jointly wrong.

That's the bet ACE makes: the same merit-order brain, the same feasibility-first gate, and the same audited actuator that dispatch requests are also what the utilization and reliability levers need. Not four products — one economic control loop, pointed at every stream your fleet emits.

---

References

1. Review of Economic Dispatch in Multi-Area Power System: State-of-the-art and Future Prospective. Electric Power Systems Research, 2023. sciencedirect.com · Background on the merit-order / economic-dispatch formulation: Merit order (overview)
2. W. Xiao, R. Bhardwaj, R. Ramjee, et al. Gandiva: Introspective Cluster Scheduling for Deep Learning. USENIX OSDI 2018. usenix.org
3. A. Qiao, S. K. Choe, S. J. Subramanya, et al. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. USENIX OSDI 2021. usenix.org
4. S. J. Subramanya, D. Arfeen, S. Lin, A. Qiao, Z. Jia, G. R. Ganger. Sia: Heterogeneity-aware, Goodput-optimized ML-cluster Scheduling. ACM SOSP 2023. doi.org/10.1145/3600006.3613175
5. Z. Yang, Z. Wu, M. Luo, W.-L. Chiang, et al. SkyPilot: An Intercloud Broker for Sky Computing. USENIX NSDI 2023. usenix.org
6. M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, F. Yang. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758