Merit-Order Workload Dispatch: Dynamic GPU Allocation for Peak Cluster Goodput

A grid operator dispatching 40 GW of load does not reason about individual power plants. They build one supply curve — every generator stacked by marginal cost, cheapest first — and fill it from the bottom until supply meets demand. Nuclear and hydro (sunk cost, ~zero marginal) run flat-out; combined-cycle gas fills the middle; peaker plants light up only for the last slice of demand. This is economic dispatch, the operating discipline of electricity markets for close to a century [1].

A fleet of in-house GPUs, reserved third-party capacity (PTU/GSU), spot, and pay-as-you-go APIs is the same problem: heterogeneous supply whose marginal costs differ by one to two orders of magnitude. Dispatched without a common ordering, the cheapest capacity — owned GPUs already paid for — sits partly idle while the most expensive rung absorbs forecastable baseline load.

ACE is merit-order economic dispatch for heterogeneous GPU + API fleets. This post covers the model and the mechanism.

The current market landscape

The two disciplines this sits between are each mature.

Economic dispatch is solved in the domain that invented it. The merit-order formulation — rank generators by ascending marginal cost, dispatch cheapest-first to meet demand at least total cost — is textbook power systems, extended in the modern literature to multi-area grids, uncertain (renewable) supply, and reserve co-optimization [1]. Real grids do not run pure merit order: they run security-constrained economic dispatch, where physical constraints (transmission limits, contingency reserves) define the feasible set first and cost optimizes only within it.

ML cluster scheduling optimizes a different variable: placement. Gandiva [2] introduced introspective, time-sliced GPU sharing and job migration. Pollux [3] reframed the objective around goodput — throughput weighted by statistical efficiency — and co-adaptively resized jobs to maximize it cluster-wide. Sia [4] extended goodput-optimized scheduling to heterogeneous clusters, matching elastic jobs to GPU type and count. This body of work answers "where should this job run inside my owned cluster?", not "given live prices across four substrates, where is the marginal token cheapest to serve now, and how much cheap capacity should have been bought last week?"

Provisioning arbitrage is a third rung. SkyPilot [5] treats the multi-cloud market as a two-sided broker, launching replicas on the cheapest available spot/on-demand capacity across providers and regions. It operates at provisioning time, not on the request hot path, and has no view of in-house fleet utilization or committed third-party blocks.

The measured gap: Microsoft's two-month Philly trace study found average GPU utilization of ~52% on a production multi-tenant cluster [6], a figure that has moved little across traces published since.

The products show the same fragmentation. LiteLLM and the API-gateway tier own provider normalization and fallback chains, but are marginal-cost-blind: they weight traffic across endpoints with no model of where each destination sits on its cost-vs-utilization curve, and no concept of an in-house fleet (3P-only). Run:ai / KAI own in-house Kubernetes GPU packing, but are 3P-blind: they optimize the supply of GPUs you own with no representation of a PTU block or a PAYG endpoint, so they cannot make the buy-vs-spill decision. Each owns one rung.

Where ACE is different

ACE treats the control loop as economic dispatch — continuously discover the supply curve, dispatch cheapest-first, safety-gate the result — across the entire heterogeneous fleet at once. Three differences:

1. One supply curve across every substrate. In-house, reserved-3P, spot, on-demand, and PAYG register as destinations behind a single uniform interface, and one marginal-cost merit order arbitrates across all of them. A 3P-only version is a gateway; an in-house-only version is a packer.
2. Security-constrained dispatch, not pure price. Capability and compliance constraints define the feasible set before cost optimizes within it — cost cannot override a data-residency rule, as a grid's transmission limits gate its dispatch.
3. The slow loop and the ledger. Merit order is the fast loop. Committed capacity is sized against forecast demand in a slow loop, and results are measured against a counterfactual by a goodput-per-dollar chargeback ledger.

The supply curve: cost curves and the stutter knee

Every destination carries a cost curve whose shape encodes its tier. An in-house GPU is baseload: already paid for, so marginal cost to the capacity ceiling is ~zero, then a hard cliff at what you own. Reserved third-party capacity behaves the same way up to its committed cap. Spot is cheap and interruptible, with a short eviction warning. PAYG is the peaker: expensive per token, effectively infinite, instantly available.

The stutter knee is the second-order term. A generator's marginal cost is not flat to its nameplate rating — pushing toward the limit costs heat rate and ramp stress. A GPU serving latency-sensitive traffic behaves the same: past some utilization point, queueing delay consumes the latency SLO, so effective marginal cost rises while dollar cost does not. ACE prices that in directly. The dispatch rule that follows: fill each tier to its stutter knee, not to its cap. A packer that fills to 100% maximizes a utilization dashboard while degrading latency SLOs.

Dispatch: rank, but constrain first

The fast loop is two steps, in this order.

Step 1 — feasibility (the security constraint). ACE filters the fleet to the set that can legally and physically serve the request: capability mask (model, VRAM class, accelerator family, interconnect for a tensor-parallel shard) and the compliance gate, which runs before merit order and which cost and latency cannot override. An EU-personal-data request is not eligible for a US-jurisdiction destination at any price. This is security-constrained economic dispatch: constraints define the feasible set, price optimizes inside it.

Step 2 — merit order. Among the survivors, ACE ranks by effective marginal cost plus two GPU-specific correction terms and picks the cheapest: a warmth term (a model already resident in VRAM is cheaper to serve than one requiring a cold load) and the latency penalty from the stutter knee. Ranking biases such as a provider-preference weight adjust where a destination sorts but never mutate the underlying cost curve — the ledger re-derives the true marginal, so a routing preference cannot inflate the savings number.

Spill: reliability is efficiency

When a destination fails — server error, timeout, connection refused — ACE trips it out of the feasible set, so the next request re-dispatches to the next-cheapest healthy destination. There is no separate failover subsystem; spill-to-next-cheapest is merit order running over a fleet with one fewer generator online, the same way a grid's supply curve shifts when a unit is lost.

The pricing view makes the unification exact. A rate-limit response means usable capacity shrank — the curve's cap moves left. A server error means effective marginal cost went to infinity — the destination drops off the merit order entirely. Reliability signals are cost-curve updates. A degrading-but-alive link is priced down rather than cut, so it sheds load gradually: the derate, not the trip.

The slow loop: unit commitment

Dispatch given existing capacity is the fast loop. A grid operator makes the larger decision hours ahead, in unit commitment: deciding which units to bring online before demand materializes, so cheap baseload is already running at the morning ramp.

The compute analog is commitment sizing. ACE's slow loop forecasts demand and sizes committed/in-house capacity to roughly the p90 of the curve, so PAYG absorbs only the tail rather than the forecastable baseline. Two related levers sit alongside it: valley-filling (moving deferrable offline work — batch indexing, evals, training — into the diurnal troughs online traffic leaves empty) and spot arbitrage (a live price feed tracking the cheapest interruptible capacity across clouds, with a proactive drain on the eviction notice so a reclaim becomes a checkpoint rather than a failure).

The ledger

A grid settles every interval against a market price. ACE's equivalent is a goodput-per-marginal-dollar ledger built from one per-request event spine. Two properties:

• Counterfactual. Every request records what a naive baseline would have cost on that exact traffic — the all-PAYG line. Savings decompose four ways: utilization, arbitrage, model-routing, caching.
• Anti-gamed. A saving counts only if served-fraction, SLO-attainment, and bounded cache false-hit rate all held. Cheaper-but-dropped or cheaper-but-degraded does not score.

Because every dispatch decision and its counterfactual land on one event spine, any claimed number traces back to the requests that produced it.

One control loop, not four tools

The same function can be assembled from parts: a gateway for provider fallback, a K8s packer for in-house, a provisioner for spot, a router for model choice. Each tool then optimizes its own rung blind to the others — the gateway weights traffic without knowing the in-house pool is past its knee; the provisioner launches a spot replica for load PAYG was already covering; the model router downgrades a request on a box that was idle. Economic dispatch is a global optimization over one supply curve; decomposed into four independent local optimizers it yields four locally-sensible decisions that are jointly suboptimal.

ACE runs one merit-order ranking, one feasibility-first gate, and one audited actuator across every stream the fleet emits.

---

References

1. Review of Economic Dispatch in Multi-Area Power System: State-of-the-art and Future Prospective. Electric Power Systems Research, 2023. sciencedirect.com · Background on the merit-order / economic-dispatch formulation: Merit order (overview)
2. W. Xiao, R. Bhardwaj, R. Ramjee, et al. Gandiva: Introspective Cluster Scheduling for Deep Learning. USENIX OSDI 2018. usenix.org
3. A. Qiao, S. K. Choe, S. J. Subramanya, et al. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. USENIX OSDI 2021. usenix.org
4. S. J. Subramanya, D. Arfeen, S. Lin, A. Qiao, Z. Jia, G. R. Ganger. Sia: Heterogeneity-aware, Goodput-optimized ML-cluster Scheduling. ACM SOSP 2023. doi.org/10.1145/3600006.3613175
5. Z. Yang, Z. Wu, M. Luo, W.-L. Chiang, et al. SkyPilot: An Intercloud Broker for Sky Computing. USENIX NSDI 2023. usenix.org
6. M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, F. Yang. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. USENIX ATC 2019. arXiv:1901.05758

---

Sign up to ACE now