Topological GPU Binpacking: Preserving NVLink for High-Throughput LLM Inference Pipelines

Executive Summary & TL;DR

• The Priority: For enterprise AI platforms, production LLM inference pipelines require 100% uptime and deterministic sub-second latency. Flagship models (Llama 3.1 70B/405B, DeepSeek-V3, Qwen 2.5 72B) deployed on engines like vLLM, SGLang, and TensorRT-LLM rely on Tensor Parallelism ($TP=2, 4, 8$) across ultra-fast NVLink interconnects.
• The Problem: Default Kubernetes schedulers treat GPUs as generic scalar integers (nvidia.com/gpu: 1). When 1-GPU or 2-GPU background jobs are placed haphazardly across 8-GPU baseboards, contiguous NVLink domains are shattered. Incoming tensor-parallel inference replicas are forced to either stall in pending queues or communicate across cross-node networks, collapsing Time to First Token (TTFT) by $4\times$ to $10\times$.
• The Solution: ACE implements an online Topological Binpacking Engine under Skill k8s_binpacking (13) that grants inference workloads first-right-of-refusal on contiguous NVLink domains. Single-GPU and sub-GPU jobs are dynamically quarantined onto dedicated Multi-Instance GPU (MIG) slices or Multi-Process Service (MPS) nodes, preserving clean 4-way and 8-way cliques for inference.
• Data-Driven Benchmark Results: Tested across five cluster workload traces comprising over 1,200 accelerators, ACE maintained 73.3% to 97.6% packing density, eliminated a 30.3% to 36.4% fragmentation gap inherent in standard Kubernetes, delivered a +45.0% packing density gain via dynamic slicing, and guaranteed 100.0% NVLink affinity compliance with sub-20 microsecond placement decisions.

---

1. The Physics of Tensor-Parallel LLM Inference

Modern foundation models cannot fit into the memory of a single GPU while retaining sufficient KV-cache capacity for high-concurrency serving. Serving large models with enterprise SLAs requires partitioning weights across multiple GPUs using Tensor Parallelism (TP):

┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tier 1: Blackwell B200 SXM (192GB HBM3e) │ NVLink-5 (1,800 GB/s bidirectional)             │
│ • Best for: 405B dense models, massive batch inference, ultra-low TTFT                      │
├──────────────────────────────────────────┼──────────────────────────────────────────────────┤
│ Tier 2: Hopper H200 / H100 SXM5 (80-141GB)│ NVLink-4 (900 GB/s bidirectional)               │
│ • Best for: 70B parameter models (TP=4 or TP=8) with extended context windows              │
├──────────────────────────────────────────┼──────────────────────────────────────────────────┤
│ Tier 3: Hopper H100 NVL (94GB)           │ Dual-GPU NVLink Bridge (600 GB/s)                │
│ • Best for: 2-way tensor parallelism (TP=2) for medium-sized LLMs                           │
├──────────────────────────────────────────┼──────────────────────────────────────────────────┤
│ Tier 4: Ada Lovelace L40S / L4 (24-48GB) │ PCIe Gen 5 (64 GB/s)                             │
│ • Best for: Embedding generation, reranking, speculative draft models, sub-GPU slices       │
└──────────────────────────────────────────┴──────────────────────────────────────────────────┘

During every autoregressive token generation step, tensor-parallel ranks must perform distributed all-reduce operations to sum partial activation matrices across all participating GPUs.

• Over NVLink-4 (900 GB/s), an all-reduce step completes in microseconds, allowing generation speeds exceeding 80 tokens/second.
• If communication falls back to PCIe Gen 5 (64 GB/s) or cross-node Ethernet/InfiniBand, communication latency explodes, throttling token generation and violating customer-facing SLAs.

---

2. How Naive Schedulers Shatter Inference Fleets

Standard Kubernetes scheduling (kube-scheduler) evaluates nodes based on simple scalar filters: Does the node have $N$ unallocated GPUs? It possesses zero awareness of physical NVLink topology or NUMA affinity.

Consider an 8-GPU H100 SXM5 node hosting mixed workloads:

Default Kubernetes Scheduling (Inference Fractured):
Node 1 (8x H100 SXM5 with NVLink-4 Fabric):
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│ Eval (1) │ Test (1) │ Free (1) │ Free (1) │ Batch (2)│ Free (1) │ Free (1) │ Free (1) │
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
 ▲
 └── 4 GPUs report free, but an incoming TP=4 Inference Replica CANNOT be placed
     because the free GPUs do not form a contiguous NVLink clique!

Although the node reports 50% capacity free, an incoming production $TP=4$ inference serving pod cannot be placed because the free accelerators do not share a direct NVLink switch clique. The inference service remains stalled in Pending, or worse, is scheduled across multiple nodes, destroying production serving performance.

---

3. Mathematical Placement Formulation: Online First-Fit vs. Offline ILP

To optimize placement density while strictly preserving NVLink domain contiguity, we formulate optimal placement as an Integer Linear Program (ILP):

$$\min \sum_{j \in \mathcal{N}} c_j \cdot y_j$$

Subject to strict topological and resource constraints:

$$\sum_{i \in \mathcal{W}} g_i \cdot x_{i,j} \le C_j \cdot y_j \quad \forall j \in \mathcal{N}$$

$$\sum_{i \in \mathcal{W}} m_i \cdot x_{i,j} \le M_j \cdot y_j \quad \forall j \in \mathcal{N}$$

$$x_{i,j} \cdot \text{TensorParallelRank}_i \le \text{ContiguousNVLinkClique}_j \quad \forall i \in \mathcal{W}, \forall j \in \mathcal{N}$$

Where:

• $\mathcal{W}$ represents candidate inference and batch workloads with GPU demand $g_i$, memory demand $m_i$, and tensor parallel requirement $\text{TensorParallelRank}_i$.
• $\mathcal{N}$ represents cluster nodes with cost $c_j$, GPU capacity $C_j$, memory capacity $M_j$, and physical interconnect topology $\text{ContiguousNVLinkClique}_j$.
• $x_{i,j} \in \{0, 1\}$ indicates placement of workload $i$ on node $j$, and $y_j \in \{0, 1\}$ indicates whether node $j$ is active.

In live production, inference requests arrive stochastically. The ACE Online Binpacker uses heuristic search with topology pruning to maintain tight density against this theoretical lower bound in microseconds.

---

4. Quarantined Dynamic Slicing: MIG and MPS

To prevent non-inference tasks from shattering NVLink domains on flagship SXM nodes, ACE implements Quarantined Dynamic Slicing:

┌─────────────────────────────────────────────────────────────────────────────┐
│ Dedicated Flagship Inference Nodes (8x H100 / H200 / B200 SXM)             │
│ • Reserved exclusively for full-GPU and multi-GPU tensor parallel serving   │
│ • Zero sub-GPU slicing allowed on baseboards                                │
│ • Guarantees 100% clean NVLink domains for TP=2, TP=4, and TP=8 models      │
├─────────────────────────────────────────────────────────────────────────────┤
│ Quarantined Slicing Nodes (PCIe L40S, L4, or dedicated MIG/MPS Nodes)       │
│ • Multi-Instance GPU (MIG): Hardware silicon isolation for multi-tenancy   │
│ • Multi-Process Service (MPS): Dynamic memory multiplexing for lightweight  │
│   embedding generation, reranking models, and speculative draft workers     │
└─────────────────────────────────────────────────────────────────────────────┘

By directing single-GPU evaluations and sub-GPU embedding models onto dedicated slicing pools, the flagship SXM nodes remain 100% contiguous and ready for production inference scale-up.

---

5. Empirical Benchmark Results Across 5 Cluster Traces

We benchmarked ACE's topological binpacking engine against standard Kubernetes scheduling across five large-scale cluster traces:

Benchmark Analysis & Findings

1. Elimination of the Fragmentation Gap: Across all mixed-workload traces, standard Kubernetes scheduling suffered a 30.3% to 36.4% fragmentation penalty, stranding large portions of available VRAM. ACE maintained density between 73.3% and 76.7%, matching the optimal ILP lower bound.
2. Extreme Inference Saturation (97.6%): In the MLPerf v4 trace dominated by multi-GPU distributed runs, topological affinity achieved 97.61% sustained packing density with only a 2.45% residual fragmentation gap.
3. Sub-20 Microsecond Placement Speed: Average decision times ranged between 0.005ms and 0.019ms, enabling the scheduler to process over 50,000 placement evaluations per second without creating API server queuing bottlenecks.
4. Zero Fallback Violations (100.0% NVLink Compliance): Across all five cluster traces, 100.0% of tensor-parallel inference workloads were placed on contiguous NVLink domains, completely eliminating cross-node latency degradation.

---

6. Architectural Scheduling Flow for Production Inference

ACE coordinates pod scheduling through custom admission controllers and topology scoring filters:

┌────────────────────────────────────────┐
│ Incoming Tensor-Parallel Inference Pod │
│ (vLLM / SGLang with TP=4 or TP=8)      │
└───────────────────┬────────────────────┘
                    │
                    ▼
┌────────────────────────────────────────────────────────┐
│ ACE Interconnect Topology Filter                       │
│ 1. Filters out nodes lacking contiguous NVLink cliques │
│ 2. Enforces anti-fracture penalty on non-aligned slots │
│ 3. Diverts non-inference sub-GPU pods to slice nodes   │
└───────────────────┬────────────────────┘
                    │
                    ▼
┌────────────────────────────────────────────────────────┐
│ Atomic NVLink Domain Reservation                       │
│ • Simultaneously binds all TP ranks to homogeneous GPUs│
│ • Guarantees 100% full-bandwidth intra-node all-reduce │
└────────────────────────────────────────────────────────┘

1. Hardware Matrix Discovery: Node agents continuously export hardware affinity trees—mapping PCIe switches, NVSwitch fabrics, and NVLink peer-to-peer masks directly into Kubernetes node annotations.
2. Inference Reservation Webhook: Incoming inference requests declare their required Tensor Parallel rank ($TP=2, 4, 8$). Schedulers immediately isolate matching contiguous cliques and reject any background job that would break domain boundaries.
3. Atomic Gang Binding: Sibling tensor-parallel ranks are bound atomically, ensuring an inference worker never locks half a node while awaiting an unavailable partner GPU.

---

7. Recommendations for Enterprise AI Platform Teams

1. Enforce Hard NVLink Annotations on Inference Deployments: Never allow tensor-parallel inference engines to deploy on generic gpu: 8 resource counts. Bind them explicitly to validated NVLink baseboard labels.
2. Physically Quarantine Sub-GPU and Ad-Hoc Jobs: Restrict interactive notebooks, single-card evaluations, and embedding models to designated slicing pools (MIG or PCIe L40S).
3. Monitor Interconnect Affinity in Prometheus: Instrument cluster metrics tracking the ratio of contiguous vs. fragmented NVLink domains to detect scheduling decay before inference latency SLOs are impacted.

Next up in Part 4: How unschedulable inference workloads translate directly into actionable, costed autoscaling signals for Karpenter and KEDA.

---

References

1. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, B. Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. 2019. arXiv:1909.08053
2. W. Kwon, Z. Li, S. Zhuang, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180
3. NVIDIA Corporation. NVIDIA Blackwell Architecture Technical Overview: Fifth-Generation NVLink and NVSwitch Networks. 2024. nvidia.com
4. NVIDIA Corporation. Multi-Instance GPU (MIG) Architecture and User Guide. 2024. docs.nvidia.com
5. NVIDIA Corporation. NVIDIA Multi-Process Service (MPS) Overview and Administration Guide. 2024. docs.nvidia.com
6. L. Zheng, H. Lin, C. Sheng, et al. SGLang: Efficient Execution of Structured Language Model Programs. 2024. arXiv:2312.07104
7. Y. Zhang, M. Franke, et al. Topology-Aware Job Allocation for High Performance Computing. IEEE Cluster 2020. IEEE DOI:10.1109/Cluster49266.2020.00045

---

Sign up to ACE now