1. Introduction: The Monolithic LLM Inference Bottleneck
Most LLM inference stacks still run every request through a single GPU pool. This means one model instance handles the entire lifecycle of a prompt — from ingestion to the final generated token. While this approach looks simple on paper, it quietly wastes a significant share of GPU capacity due to how differently the two phases of inference behave.
- Prefill Phase (The Compute Bottleneck): The model processes the entire input prompt in parallel, computing attention across every token at once. Dominated by large matrix multiplications (GEMMs), this is extremely compute-heavy. It directly determines Time to First Token (TTFT).
- Decode Phase (The Memory Bottleneck): The model generates output tokens one at a time, autoregressively. It repeatedly reads the entire KV cache from GPU memory. This is a memory-bandwidth-bound operation and determines Time Per Output Token (TPOT) and Inter-Token Latency (ITL).
When both phases share the same GPU, they interfere with each other. A long prefill burst stalls decode steps, spiking ITL for current users. Conversely, a batch full of decode steps leaves compute units under-utilized while new requests wait, increasing TTFT.
Prefill-Decode (PD) Disaggregation solves this by physically separating the two phases onto different GPU pools.
| Feature | Monolithic Inference | Disaggregated Inference |
|---|---|---|
| GPU Usage | Single shared pool | Split dedicated pools |
| Prefill Workload | Competes with Decode steps | Runs purely on Compute-heavy GPUs |
| Decode Workload | Competes with Prefill bursts | Runs purely on Memory-heavy GPUs |
| Primary Bottleneck | Compute & Memory clash | Network transfer (KV Cache) |
2. Architecture Overview: The H100 + A100 Synergy
Splitting prefill and decode pays off when each pool is matched to hardware that fits its specific bottleneck. The H100 and A100 combination is the industry standard for this pattern:
- NVIDIA H100 for Prefill: The H100's FP8/FP16 throughput and Tensor Core design chew through large GEMM operations instantly, compressing TTFT.
- NVIDIA A100 80GB for Decode: Decode needs massive memory capacity and bandwidth to hold and scan KV caches across concurrent sessions. The A100 delivers this at a materially lower cost per GPU-hour than an H100.
Workflow Architecture
- Routing: The client sends a prompt to a routing layer in front of the Kubernetes cluster.
- Prefill (H100): The router forwards the prompt to an available worker in the Prefill Pool.
- KV Transfer: The H100 processes the prompt, builds the KV cache, and streams it to the Decode Pool over a high-bandwidth interconnect (RDMA/RoCE v2).
- Decode (A100): The A100 worker takes over autoregressive generation, streaming output tokens back to the client.
3. Prerequisites & Infrastructure Requirements
Before deploying a disaggregated setup, confirm your bare-metal or cloud cluster meets these baselines:
- Kubernetes v1.28+: Required for advanced node taint/toleration and topology-aware scheduling.
- NVIDIA GPU Operator: Installed and configured with RDMA/InfiniBand device plugin support.
- High-Bandwidth Network Fabric: 100Gbps+ RoCE v2 or InfiniBand connecting the prefill and decode node pools.
- Dedicated GPUs: Assumes dedicated (non-shared/non-MIG) GPU allocation per worker pod.
4. Step-by-Step Implementation Guide
Step 1: Labeling and Tainting Kubernetes GPU Nodes
Start by tagging nodes according to their GPU architecture, then taint them so the scheduler won't place unrelated workloads on these specialized nodes.
# Label and Taint H100 nodes for the prefill role
kubectl label nodes h100-node-01 gpu-role=prefill gpu-type=h100
kubectl taint nodes h100-node-01 role=prefill:NoSchedule
# Label and Taint A100 nodes for the decode role
kubectl label nodes a100-node-01 gpu-role=decode gpu-type=a100
kubectl taint nodes a100-node-01 role=decode:NoScheduleStep 2: Deploying the Disaggregation Engine & KV Cache Router
Deploy the routing layer that sits between the client and both pools. This proxy knows how to reach both pools and route KV cache transfer requests.
apiVersion: v1
kind: ConfigMap
metadata:
name: pd-router-config
namespace: llm-inference
data:
routing-policy.yaml: |
prefill_pool:
service: prefill-service.llm-inference.svc.cluster.local
port: 8000
decode_pool:
service: decode-service.llm-inference.svc.cluster.local
port: 8001
kv_transfer:
backend: rdma
timeout_ms: 200Step 3: Provisioning the Prefill GPU Pool (H100)
The prefill deployment targets H100 nodes. It stops after producing the first token and generates the KV cache. Notice the /dev/shm mount, which acts as a massive staging buffer for the KV cache before network handoff.
apiVersion: apps/v1
kind: Deployment
metadata:
name: prefill-worker
namespace: llm-inference
spec:
replicas: 2
selector:
matchLabels:
app: prefill-worker
template:
metadata:
labels:
app: prefill-worker
spec:
nodeSelector:
gpu-role: prefill
tolerations:
- key: "role"
operator: "Equal"
value: "prefill"
effect: "NoSchedule"
containers:
- name: vllm-prefill
image: vllm/vllm-openai:latest
args:
- "--model=meta-llama/Llama-3.1-70B-Instruct"
- "--disaggregation-mode=prefill"
- "--kv-transfer-backend=rdma"
resources:
limits:
nvidia.com/gpu: 1
volumeMounts:
- name: shm
mountPath: /dev/shm
volumes:
- name: shm
emptyDir:
medium: Memory
sizeLimit: "16Gi"Step 4: Provisioning the Decode GPU Pool (A100)
The decode deployment targets A100 nodes. You typically need a higher replica count here because decode workers hold sessions open for the full length of the text generation.
apiVersion: apps/v1
kind: Deployment
metadata:
name: decode-worker
namespace: llm-inference
spec:
replicas: 4
selector:
matchLabels:
app: decode-worker
template:
metadata:
labels:
app: decode-worker
spec:
nodeSelector:
gpu-role: decode
tolerations:
- key: "role"
operator: "Equal"
value: "decode"
effect: "NoSchedule"
containers:
- name: vllm-decode
image: vllm/vllm-openai:latest
args:
- "--model=meta-llama/Llama-3.1-70B-Instruct"
- "--disaggregation-mode=decode"
- "--kv-transfer-backend=rdma"
- "--tensor-parallel-size=2"
resources:
limits:
nvidia.com/gpu: 2
volumeMounts:
- name: shm
mountPath: /dev/shm
volumes:
- name: shm
emptyDir:
medium: Memory
sizeLimit: "16Gi"5. Network Optimization for KV Cache Transfer
KV cache transfer makes or breaks a disaggregated setup. Transferring gigabytes of KV cache over standard TCP/IP introduces serialization overhead and kernel-space copying that destroys latency gains.
RDMA (Remote Direct Memory Access) avoids this by moving data directly between GPU memory on different nodes. On Kubernetes, this requires:
- Enabling RoCE v2 or InfiniBand through the NVIDIA network operator.
- Using an RDMA-aware CNI plugin (e.g., SR-IOV).
- Requesting the RDMA device explicitly in your pod resources (e.g.,
rdma/hca: 1).
6. Benchmarking & Performance Results
Validate your split-pool setup against your monolithic baseline using tools like vllm-benchmark-client.
- Run identical workloads: Push the same prompt/output distribution to both setups.
- Track TTFT and TPOT separately: Disaggregation should visibly compress TTFT under concurrent load.
- Measure aggregate throughput: Track tokens/sec/GPU across both pools combined.
Published deployments report TTFT reductions of up to 3x under heavy concurrent load, along with meaningfully higher aggregate GPU utilization.
7. Common Pitfalls & Troubleshooting
Network Saturation
If KV cache transfer time exceeds the compute time saved, disaggregation fails. Watch RDMA transfer latency as a first-class metric. If it creeps up, investigate your network fabric or NIC allocation.
Unbalanced Pools
Prefill scales with request rate and prompt length. Decode scales with concurrent session count and generation length. Autoscaling policies must key off pool-specific signals (e.g., queue depth for prefill, active sessions for decode) rather than a shared CPU metric.
Router as a Single Point of Failure
Every request passes through the routing layer. It requires multiple replicas behind a Kubernetes Service and independent health checks against both pools.
KV Cache Format Mismatches
If prefill and decode workers run different versions of the inference engine, KV cache serialization formats will drift out of sync, causing silent failures. Keep engine images strictly version-matched.
Sizing the Prefill-to-Decode Ratio
Do not blindly copy the 2:1 ratio used in the manifests above. Divide your average prompt length by your average generation length to map your exact workload. Chatbots need a decode-heavy ratio; document summarization needs a prefill-heavy ratio.
Cold-Start Latency on Scale-Up
Autoscaling a decode pool doesn't help instantly — a new A100 pod must load model weights first. Pre-warm a small buffer of idle decode replicas to bridge this gap.
8. Conclusion & Next Steps
Prefill-decode disaggregation fixes a structural mismatch: one phase is compute-bound, the other is memory-bound. Splitting them across H100s (Prefill) and A100s (Decode) over RDMA lets each pool scale around its actual bottleneck. It is currently one of the most cost-effective ways to scale enterprise LLM hosting on Kubernetes.
To move forward, benchmark your own model's traffic pattern against a monolithic baseline, tune the prefill-to-decode replica ratio, and implement autoscaling based on stable per-pool load signals.