Utilization headroom governor

Maintains fleet load within target bands to ensure headroom for sudden bursts.

problem it solves
Prevents full GPU capacity saturation that leads to sudden latency spikes during load surges.

What it does

What it does: Holds fleet-wide GPU utilization within a target band (e.g. 65%-80%) over multi-minute windows.

What it watches: Fleet average GPU compute and memory utilization over rolling load history.

When it triggers: When fleet load drifts outside configured target headroom boundaries.

The Action: Adjusts autoscaler targets smoothly to ensure headroom for incoming traffic bursts.

How it recovers: Nudges capacity targets back down as sustained baseline traffic subsides.

GPU autoscaling skill enabledrequired
Requires GPU autoscaling to execute capacity target adjustments.
At least 7 days of load historyrecommended
Provides baseline data for historical headroom modeling.

Mode	Effect on your request	What you can see
off	Not consulted. Autoscaler operates purely on short-term reactive metrics.	No utilization_headroom stage recorded.
shadow	Calculates headroom targets and logs recommended governor adjustments.	Stage with action=would_adjust_headroom logged.
prod	Applies smooth target adjustments to autoscaler control loops.	Target band occupancy and headroom adjustments logged.

Target load band	65% - 80% fleet utilization	Optimal balance of cost and burst safety.
Adjustment window	5-minute smoothed moving average	Prevents short-term metric noise.
Safety buffer	20% immediate headroom floor	Guarantees capacity for sudden spikes.

·Running at lower target utilization (e.g. 60%) increases standing infrastructure cost for higher availability.
·Governor updates occur over minutes to avoid thrashing autoscaler provisioning.
·Requires historical load data to optimize headroom bands accurately.