The Compute Efficiency Audit: Sizing PTU Commitments with Critical Fractile Mathematics

When managing LLM inference infrastructure at scale, engineering and FinOps teams encounter a fundamental tradeoff across compute tiers: Pay-As-You-Go (PAYG) vs. Provisioned Throughput (Azure PTUs, AWS Bedrock Model Units, Google Cloud Vertex GSUs, and In-House GPU Clusters).

• On-Demand PayGo: Offers infinite elastic scaling with zero commitment, but charges a 40%–60% pricing premium on every processed token.
• Committed Provisioning: Provides deep unit cost discounts, but introduces fixed sunk costs where underutilized capacity degrades effective cost per token.

Sizing commitments with naïve heuristics—such as provisioning for peak concurrency or sizing strictly for 24-hour average throughput—inevitably leads to either severe over-provisioning (stranded nocturnal capacity) or under-provisioning (catastrophic queue backpressure and SLA violations during traffic surges).

The ACE Compute Efficiency Audit frames committed capacity procurement as a continuous mathematical optimization problem. Below is the exact formulation, derived from stochastic inventory theory, to determine the optimal provisioned commitment floor.

---

1. Mathematical Formulation: Sunk-Cost Arbitrage & The Critical Fractile

Committed GPU capacity sizing can be modeled as a continuous Newsvendor Problem under stochastic diurnal demand.

Let $D$ denote the random variable representing token demand per second across a 24-hour period, characterized by cumulative distribution function $F(x) = P(D \le x)$ and probability density function $f(x)$.

Let $C$ denote the committed throughput capacity in tokens per second ($\text{tok/s}$).

Every token routed incurs costs according to two operational states:

1. Underage Penalty ($c_{\text{under}}$): The marginal cost incurred when demand exceeds provisioned capacity ($D > C$). Tokens that cannot be served on the committed baseline must spill over into elastic PayGo endpoints at list price ($p_{\text{payg}}$) rather than the lower committed amortized unit price ($p_{\text{ptu}}$): $$c_{\text{under}} = p_{\text{payg}} - p_{\text{ptu}}$$

2. Overage Penalty ($c_{\text{over}}$): The marginal cost incurred when provisioned capacity sits idle ($D < C$). Because committed capacity is pre-paid and non-refundable, unutilized tokens represent lost capital: $$c_{\text{over}} = p_{\text{ptu}}$$

Expected Total Cost Function

The expected cost function $E[T(C)]$ over a given billing window is expressed as:

$$E[T(C)] = c_{\text{over}} \int_{0}^{C} (C - x) f(x) , dx + c_{\text{under}} \int_{C}^{\infty} (x - C) f(x) , dx$$

Differentiating $E[T(C)]$ with respect to $C$ using Leibniz's Rule and setting the derivative to zero yields:

$$\frac{d}{dC} E[T(C)] = c_{\text{over}} F(C) - c_{\text{under}} (1 - F(C)) = 0$$

$$(c_{\text{over}} + c_{\text{under}}) F(C^*) = c_{\text{under}}$$

$$F(C^) = P(D \le C^) = \frac{c_{\text{under}}}{c_{\text{under}} + c_{\text{over}}}$$

Substituting $c_{\text{under}} = p_{\text{payg}} - p_{\text{ptu}}$ and $c_{\text{over}} = p_{\text{ptu}}$:

$$F(C^*) = \frac{p_{\text{payg}} - p_{\text{ptu}}}{(p_{\text{payg}} - p_{\text{ptu}}) + p_{\text{ptu}}} = \frac{p_{\text{payg}} - p_{\text{ptu}}}{p_{\text{payg}}} = 1 - \frac{p_{\text{ptu}}}{p_{\text{payg}}}$$

The Critical Fractile Rule of LLM Provisioning:

$$\text{Optimal Baseload Quantile } q^* = 1 - \frac{p_{\text{ptu}}}{p_{\text{payg}}} = \text{Discount Percentage vs. PAYG}$$

Probability Density f(x)
 ▲
 │           Diurnal Demand Distribution
 │                  ╭──────────╮
 │                ╭─╯          ╰─╮
 │              ╭─╯              ╰─╮
 │             ╭╯                  ╰╮
 │           ╭─╯                    ╰─╮
 │         ╭─╯                        ╰─╮
 │       ╭─╯                            ╰─╮
 └───────┴─────────────────┬──────────────┴───────► Demand (tok/s)
         ◄── Baseload C* ──►◄── Spillover ──►
           Area = q*          Area = 1 - q*
           (Sunk-Cost)        (Elastic PAYG)

Formal Result: If a provider contract offers a 45% effective discount on 1-year committed throughput over PayGo list prices, the optimal commitment baseload $C^*$ is precisely the 45th percentile ($p45$) of your empirical 24-hour demand distribution. Provisioning below $p45$ unnecessarily pays PayGo premiums; provisioning above $p45$ generates stranded capital during nocturnal demand troughs.

---

2. Worked Empirical Example: Enterprise 500M Tokens/Day

Consider an enterprise AI workload with the following parameters:

• Daily Traffic Volume: 500,000,000 tokens / day ($500\text{M tok/day}$).
• Peak-to-Trough Ratio: $3:1$ (diurnal follow-the-sun business chat pattern).
• Peak Throughput: $10,000\text{ tok/s}$ ($12:00\text{--}16:00\text{ UTC}$).
• Trough Throughput: $3,333\text{ tok/s}$ ($00:00\text{--}06:00\text{ UTC}$).
• Current Spend (100% PayGo): $$100,000\text{ / month}$ at blended $$0.0067 / 1\text{k tokens}$.
• Committed Rate Card: 1-year reservation at $45%$ effective discount over PayGo.

Tokens/s
 ▲
 │                 Peak Traffic (12:00 - 16:00 UTC)
10k│                     ┌───────────┐  ▲ Elastic PayGo Spillway (35% volume)
 │                     │           │  │ (Absorbs bursts with zero queuing latency)
 8k│                 ┌─┘           └─┐▼
───┼─────────────────┼───────────────┼─────────────────────── Optimal Baseload: C* = 5,621 tok/s
 6k│  ┌──────────────┘               └─────────────┐ ▲
 │  │                                             │ │ 100% Sunk-Cost Baseload (65% volume)
 4k│  │                                             │ │ (Runs at 80% knee capacity ceiling)
 │  │                                             │ ▼
 2k│──┘                                             └──
 00:00             06:00           12:00           18:00          24:00 (UTC)

Quantitative Architecture Breakdown:

Conclusion: By sizing commitments to the critical fractile ($C^* = 5,621\text{ tok/s}$) and routing excess traffic to an elastic PayGo spillway, the organization achieves $31.5%$ net monthly savings ($$377,520\text{ annualized}$) while eliminating nocturnal capacity waste and preserving $99.9%$ SLOs.

---

3. The 4 Pillars of the Tier 2 Continuous Compute Audit

In production environments, capacity planning is not a one-time static calculation. Workload distributions evolve continuously as new autonomous agents, background pipelines, and user features deploy.

The ACE Compute Efficiency Audit integrates four automated control-loop dimensions to maintain optimal fleet efficiency:

┌─────────────────────────────────────────────────────────────────────────────────┐
│                          ACE COMPUTE EFFICIENCY AUDIT                           │
├────────────────────────┬────────────────────────┬───────────────────────────────┤
│ CARD A: Committed PTU  │ CARD B: Stranded VRAM  │ CARD C: Diurnal Trough        │
│ Arbitrage              │ Reclaim                │ Capacity                      │
│ • Quantile demand fit  │ • Zero-load inference  │ • 00:00–06:00 UTC valley fill │
│ • Sunk-cost baseload   │   replica detection    │ • Batch evals / distillation  │
│ • PayGo spill routing  │ • Unlocks idle VRAM    │ • Zero-cost compute reclaim   │
├────────────────────────┴────────────────────────┴───────────────────────────────┤
│ CARD D: Optimization Skills Headroom & Shadow Counterfactuals                   │
│ • Semantic Caching • Dynamic Model Routing • Prompt & Trajectory Compaction     │
│ • Measured token-level savings before executing live on traffic                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Pillar A: Committed Sizing & PTU Arbitrage

Continuously executes empirical Newsvendor calculations (CommitmentPlanner.plan()) over rolling 7-day and 30-day token demand distributions to recommend exact reservation unit purchases and lease renewals.

Pillar B: Stranded VRAM & Silicon Leakage Reclaim

In-VPC telemetry collectors monitor accelerator allocations across private clusters (vLLM, SGLang, TensorRT-LLM) using sustained_stranded(). It flags GPU memory allocated to inactive container replicas with zero invocation throughput for $>60\text{ minutes}$, presenting privacy-masked reclaim actions to operators.

Pillar C: Diurnal Trough & Valley-Fill Planning

Evaluates nighttime compute valleys via valley_fill_plan(). Rather than letting committed GPU-hours sit idle between $00:00\text{ and }06:00\text{ UTC}$, it quantifies available off-peak GPU-hours (e.g., $340\text{ GPU-hours/day}$) for background workloads such as offline synthetic data generation, model distillation, and asynchronous batch evaluations.

Pillar D: Optimization Skills Headroom in Shadow Mode

Evaluates data-path optimization skills—including Semantic Caching, Intent-Based Routing, and Context Compaction—in zero-risk Shadow Mode. ACE measures counterfactual token and dollar reductions directly on caller traffic before promotion to active enforcement.

---

4. Run the Zero-Credential Sizing Calculator

Evaluate your organization's commitment arbitrage and optimal baseload sizing directly in your browser:

👉 Launch the Interactive Sizing Calculator on acefleet.dev/pricing

Inputs required: Monthly spend, daily token volume, peak-to-trough ratio, and committed provider discount.

---

5. Enable Full Fleet Compute Auditing

To deploy the Tier 2 Continuous Compute Audit across your multi-cloud and on-premise GPU fleets with automated headroom monitoring, connect your endpoints to the ACE Control Plane:

🚀 Deploy ACE & Trigger Your Compute Efficiency Audit