Deploying SGLang with RadixAttention on Dedicated GPU Servers: Step-by-Step Guide

Quick Summary / Key Takeaways

• SGLang is an open-source LLM serving engine built by the SGLang project (originating from LMSYS/UC Berkeley research), designed for high-throughput, low-latency inference.
• RadixAttention stores KV cache states in a radix tree, letting SGLang reuse shared prefixes (system prompts, chat history) instead of recomputing them — this is what drives its TTFT advantage in agentic and RAG workloads.
• Deployment is Docker-based: you need NVIDIA drivers, CUDA, and the NVIDIA Container Toolkit before running the SGLang server container.
• SGLang exposes an OpenAI-compatible API, so existing OpenAI SDK code can point at your dedicated GPU server with minimal changes.

Why SGLang Is Gaining Ground in LLM Serving

SGLang is an open-source serving framework for large language models, originally developed out of the LMSYS research group at UC Berkeley and now maintained under the SGLang project on GitHub. It was designed around a simple observation: a large share of real-world LLM traffic — chatbots, RAG systems, agents — repeatedly reuses the same prompt prefixes.

Instead of treating every request as an isolated computation, SGLang's runtime is built to detect and reuse that shared context automatically. This is where it tends to pull ahead of engines like vLLM specifically in agentic and multi-turn workloads, where the same system prompt or conversation history gets sent over and over.

For teams running models on a dedicated GPU server, this matters directly: better cache reuse means more requests served per GPU-hour, and lower latency on the metric users actually notice — the time before the first token appears.

Understanding SGLang and RadixAttention (The Core Mechanism)

SGLang's serving engine is built around two connected ideas: a flexible front-end for describing generation programs, and a backend runtime optimized around RadixAttention, its KV cache management strategy.

How RadixAttention Reuses KV Cache

In a standard transformer serving setup, the key-value (KV) cache for a request is discarded once the response is generated. If the next request shares the same system prompt or few-shot examples, the engine still has to recompute the attention states for that shared prefix from scratch. That recomputation is pure waste — the tokens haven't changed.

RadixAttention solves this by organizing all cached KV states across requests into a radix tree, a data structure where common prefixes are stored once and shared across branches. When a new request arrives, SGLang's scheduler walks the tree to find the longest matching prefix already in cache, and only computes attention for the new, non-overlapping tokens.

This has a direct, practical effect:

• Shared system prompts across many users are computed once, not per-request.
• Multi-turn conversations reuse the KV cache from earlier turns instead of reprocessing the entire chat history.
• Time-to-First-Token (TTFT) drops sharply on any "warm" request that shares a prefix with something already cached, because the prefill stage has less new work to do.

The tree also handles eviction automatically using an LRU-style policy, so cache memory is reclaimed for less-recently-used branches once VRAM pressure builds up. You don't manage this manually — it's part of the scheduler.

Step-by-Step SGLang Docker Deployment Guide

With prerequisites in place, deployment itself is fast. This is the core, repeatable workflow for getting SGLang running on any dedicated GPU server.

Step 1: Install NVIDIA Container Toolkit

Docker needs explicit GPU passthrough support. On Ubuntu, install and configure the toolkit like this:

# Configure the repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Configure Docker to use the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU access from inside a container before moving on:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If nvidia-smi output appears inside the container, Docker has GPU access and you're ready for SGLang.

Step 2: Launch the SGLang Server via Docker

Pull and run the official SGLang server image, pointing it at a model — here, Llama-3.1-8B-Instruct:

docker run --gpus all \
  --ipc=host \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=your_huggingface_token" \
  lmsysorg/sglang:v0.5.12 \
  python3 -m sglang.launch_server \
  --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --mem-fraction-static 0.85

A note on the image tag: this guide pins the image to v0.5.12 rather than :latest. SGLang ships new releases frequently, and :latest can point to a different build tomorrow than it does today — a real risk for a published tutorial, since a reader following these exact steps months from now could hit a breaking change or a different default behavior. Always check the SGLang releases page for the current stable tag before deploying, and pin production containers to a specific version rather than tracking :latest.

Pro tip — what those flags actually do:

• --mem-fraction-static: controls what fraction of GPU memory SGLang reserves upfront for model weights and the static KV cache pool. Set it too high and you risk out-of-memory errors when activations spike; set it too low and you underuse available VRAM.
• --ipc=host: shares the host's IPC namespace with the container. SGLang relies on shared memory for efficient inter-process communication (especially with multiple GPU workers), and Docker's default IPC namespace is often too small — this flag prevents silent crashes or hangs under load.

Note on --shm-size: don't combine it with --ipc=host. Once --ipc=host is set, the container shares the host's actual /dev/shm rather than getting its own private tmpfs, so --shm-size (which only sizes a container-private mount) has nothing left to act on and is silently ignored — no error, no warning, it just does nothing. If you need to confirm available shared memory, check the host's own capacity instead (df -h /dev/shm), not a container flag.

Once the container is running, SGLang exposes an OpenAI-compatible endpoint at http://<your-server-ip>:30000/v1, so existing OpenAI SDK client code works with only a base_url change.

Benchmarking RadixAttention: Testing TTFT Latency

The clearest way to see RadixAttention working is to send a long, shared system prompt followed by different follow-up questions, and compare TTFT — not total response time — between a cold-cache request and a warm-cache request.

Three things matter for this test to actually measure what it claims to measure:

• Stream the response (stream=True) and time the arrival of the first chunk, not the full completion. With stream=False, the client only gets a response after generation finishes, so the timer captures prefill plus the entire decode phase — that's total latency, not TTFT, and it dilutes the exact effect you're trying to isolate.
• Send a warm-up request first, and don't include it in your measurements. A freshly started server pays a one-time cost for CUDA kernel compilation and CUDA graph capture on its very first inference call — that overhead has nothing to do with RadixAttention, but it will inflate your "cold cache" number if you don't burn it off first.
• Run multiple samples and average them. GPU inference timing has natural run-to-run jitter. A single cold-vs-warm comparison (n=1) can be misleading; a small loop with an average or median is far more defensible.

Python Script for TTFT Testing

import time
import uuid
import statistics
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="none")

def build_system_prompt(unique_tag=""):
    """Builds a long system prompt. unique_tag makes the prefix unseen by the cache."""
    return (
        f"You are a technical support assistant. Reference ID: {unique_tag}. "
        "Here is the full product documentation you must reference: "
        + ("Section content. " * 500)
    )

def measure_ttft(system_prompt, question):
    """Streams the response and returns time-to-first-token in seconds."""
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
        max_tokens=50,
        stream=True,
    )
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            return time.perf_counter() - start  # time of first content token
    return None

# --- Warm-up: a SHORT, UNRELATED prompt, just to absorb one-time CUDA kernel ---
# --- compile / graph capture cost. It must NOT share a prefix with the test. ---
measure_ttft("You are a helpful assistant.", "Say hello.")

# --- Cold-cache samples: a fresh, never-before-seen prefix on every single run ---
# --- (unique_tag guarantees no radix-tree hit is possible) ---
cold_samples = [
    measure_ttft(build_system_prompt(unique_tag=str(uuid.uuid4())), "What is the warranty period?")
    for _ in range(5)
]

# --- Warm-cache samples: prime the cache ONCE with a fixed prefix (untimed), ---
# --- then send timed requests that reuse that exact same prefix ---
warm_prefix = build_system_prompt(unique_tag="warm-test-fixed-id")
measure_ttft(warm_prefix, "Priming request, ignore this result.")  # untimed, populates cache

warm_samples = [
    measure_ttft(warm_prefix, "How do I reset the device?")
    for _ in range(5)
]

print(f"Cold-cache TTFT: avg={statistics.mean(cold_samples):.3f}s, "
      f"median={statistics.median(cold_samples):.3f}s")
print(f"Warm-cache TTFT: avg={statistics.mean(warm_samples):.3f}s, "
      f"median={statistics.median(warm_samples):.3f}s")

What to expect: each cold-cache sample uses a unique, never-cached prefix, so every one of those 5 runs pays the full prefill cost — there's no cross-contamination between samples. Each warm-cache sample reuses the exact same prefix, already primed into the radix tree by the untimed priming call, so all 5 of those runs should hit the cache and show measurably lower TTFT. Exact numbers depend on your GPU, model, and prompt length, so treat this script as a repeatable test to run on your own hardware rather than a source of fixed benchmark figures. Note that time.perf_counter() still includes network round-trip time between client and server; that's negligible on localhost but worth keeping in mind if you're benchmarking a remote dedicated server over a network connection.

Advanced SGLang Configurations for Production

Once the basic deployment works, production workloads usually need more control over memory and multi-GPU scaling.

Multi-GPU Tensor Parallelism (--tp)

For models too large for a single GPU (32B, 70B+), SGLang supports tensor parallelism, splitting the model's weight matrices across multiple GPUs:

python3 -m sglang.launch_server \
  --model-path meta-llama/Meta-Llama-3-70B-Instruct \
  --tp 4 \
  --mem-fraction-static 0.85

Here, --tp 4 shards the model across 4 GPUs, with each GPU holding a fraction of every layer and synchronizing during the forward pass. This is why --ipc=host and adequate interconnect (NVLink where available) matter — tensor parallelism is communication-heavy.

FP8 and AWQ Quantization

To reduce VRAM footprint and increase throughput, SGLang supports serving quantized model checkpoints, including FP8 and AWQ:

python3 -m sglang.launch_server \
  --model-path <awq-quantized-model-path> \
  --quantization awq

Quantization trades a small amount of model precision for a significantly smaller memory footprint, which lets you serve larger models on fewer GPUs, or increase batch size and concurrent request capacity on the same hardware.

Troubleshooting Common SGLang Errors

How to Fix "CUDA Out of Memory" in SGLang

This is the most common deployment error, and it almost always traces back to memory allocation, not a broken install. Fix it by:

• Lowering --mem-fraction-static (e.g., from 0.9 to 0.8 or 0.7) to leave more headroom for activation memory and CUDA overhead.
• Reducing --max-total-tokens or context length if you're serving very long prompts.
• Confirming the model actually fits on your GPU(s) at your chosen quantization level — cross-check against the VRAM table earlier in this guide.

Port Binding and IPC Host Issues

If the container starts but crashes or hangs unpredictably under concurrent load, missing --ipc=host is a common cause. SGLang's multi-process architecture (especially with --tp > 1) depends on shared memory for inter-process communication; Docker's default IPC namespace size is often insufficient, leading to silent failures rather than a clear error message. Always include --ipc=host (or an explicitly large --shm-size) in production deployments.

The Future: Prefill-Decode Disaggregation in SGLang

A newer architectural direction in high-scale LLM serving — supported in ongoing SGLang development — is Prefill-Decode (PD) disaggregation. LLM inference has two phases with very different resource profiles: prefill (processing the input prompt, compute-heavy) and decode (generating tokens one at a time, memory-bandwidth-heavy).

Running both phases on the same GPU means one phase's resource pattern interferes with the other's. PD disaggregation splits these two stages onto separate GPU servers — dedicated prefill nodes and dedicated decode nodes — communicating over high-speed interconnects such as RDMA to transfer KV cache states between them.

For large-scale, multi-server deployments, this lets each GPU pool be sized and tuned for exactly one workload type, improving overall cluster efficiency rather than forcing every GPU to compromise between two very different jobs.

Conclusion

SGLang combined with RadixAttention gives you a serving stack that's genuinely built around how real LLM traffic behaves — repeated prefixes, multi-turn context, and agentic workflows — rather than treating every request as a blank slate. On a properly provisioned dedicated GPU server, that translates into lower TTFT, better cache utilization, and more efficient use of your hardware.

For the latest flags, model support, and architecture updates, the official SGLang GitHub repository is the authoritative source — the project moves quickly, and the documentation there reflects the current release.

If you'd rather skip the server provisioning and driver setup entirely, GPUYard offers dedicated GPU servers pre-suited for LLM serving workloads like SGLang — so you can go straight from docker run to a production endpoint without managing the underlying hardware.

Frequently Asked Questions (FAQs)

• Q: Is SGLang faster than vLLM?
  A: SGLang tends to outperform vLLM specifically in workloads with long, shared contexts — such as RAG pipelines, agentic AI, and multi-turn chat — because RadixAttention reuses cached prefixes instead of recomputing them. For single, independent requests with no shared context, the performance gap narrows.
• Q: Does SGLang support the OpenAI API format?
  A: Yes. SGLang exposes an OpenAI-compatible API endpoint, so it functions as a drop-in replacement for applications already built on the OpenAI SDK — you typically only need to change the base_url.
• Q: How much VRAM do I need for an 8B model in SGLang?
  A: Typically 16–24 GB, depending on context window length and how much VRAM you allocate to the KV cache via --mem-fraction-static. An RTX 4090 or A100 (40GB) comfortably handles most 8B deployments.
• Q: How do I enable RadixAttention in SGLang?
  A: RadixAttention is built into SGLang's scheduler by default — there's no separate flag to turn it on. Any request sharing a prefix with a cached request automatically benefits from radix tree reuse.
• Q: What causes "CUDA Out of Memory" errors in SGLang, and how do I fix them?
  A: It's usually caused by --mem-fraction-static being set too high relative to available VRAM. Lower the value, reduce max context length, or apply quantization (FP8/AWQ) to free up memory.
• Q: Why is --ipc=host required when running SGLang in Docker?
  A: SGLang relies on shared memory for inter-process communication, particularly with tensor parallelism across multiple GPUs. Docker's default IPC namespace is too restrictive for this, and omitting --ipc=host can cause silent crashes under load.