The demand for fast, affordable Large Language Model (LLM) inference is at an all-time high. Every additional millisecond of latency and every extra dollar per million tokens directly impacts product economics. To maximize throughput and lower costs, enterprise infrastructure teams are standardizing on the two most proven, scalable, and immediately available GPU architectures on the market: the NVIDIA H100 (Hopper) and the RTX Pro 6000 (Ada Lovelace).
However, having the right raw silicon is only half the battle. To extract maximum ROI from these GPUs, you need a highly optimized software stack. That engine is TensorRT-LLM, NVIDIA's open-source library for compiling and serving large language models at production scale.
This tutorial walks through the exact steps required to deploy a large model — using Llama 3 as the working example — on H100 and RTX Pro 6000 hardware with TensorRT-LLM, from pulling the correct container to serving live inference requests via an API.
Why H100 & RTX 6000 + TensorRT-LLM Is a Game Changer
FP8 Precision (Native Hardware Acceleration)
Both the H100 (Hopper architecture) and the RTX Pro 6000 (Ada Lovelace architecture) feature native support for FP8 (8-bit floating point) quantization via their fourth-generation Tensor Cores.
Cutting weight precision from FP16 to FP8 roughly halves the memory footprint of a model's weights. This has two massive compounding effects for inference:
- Higher Model Density: More of the model fits in GPU memory. For instance, a quantized Llama 3 70B can fit comfortably across fewer GPUs, drastically cutting hardware requirements.
- Expanded Paged KV Cache: More VRAM headroom is left for the KV cache, which allows a single server to sustain much larger batch sizes and process much longer context windows simultaneously.
In-Flight Batching
Real-world LLM traffic doesn't arrive in neat, uniform batches. TensorRT-LLM features in-flight batching (also called continuous batching). It evaluates the request queue at every generation step. As soon as one user's request finishes, a new one is instantly inserted into the batch — meaning your GPU compute is never left sitting idle waiting for a slower request to finish.
Lower Cost Per Token
Because GPU ownership or rental cost is fixed per hour, generating more tokens per hour directly lowers your effective cost per million tokens. The combination of FP8 speed and in-flight batching allows the H100 and RTX Pro 6000 to deliver enterprise-grade generation speeds at a fraction of the cost of naive implementations.
Hardware & Software Prerequisites
Before starting the deployment, confirm the following:
GPU Hardware (Available on GPUYard)
You will need an NVIDIA Hopper or Ada generation GPU to take full advantage of FP8 Tensor Cores.
- NVIDIA H100 Servers: The ultimate data center powerhouse, ideal for massive MoE models (like DeepSeek or Mixtral) and high-concurrency production endpoints.
- NVIDIA RTX Pro 6000 Servers: A highly cost-effective 48GB VRAM powerhouse, perfect for single-GPU inference (Llama 3 8B), AI agent workflows, and dev/test environments.
👉 Both configurations are available for immediate, bare-metal provisioning at GPUYard.
Operating System & Drivers
- Ubuntu 22.04 LTS or 24.04 LTS (Linux is the supported platform for TensorRT-LLM containers).
- A current NVIDIA driver (e.g., v535 or newer) that supports your target CUDA toolkit version.
Verify your GPU and driver are visible:
nvidia-smiDocker & NVIDIA Container Toolkit
TensorRT-LLM ships as prebuilt NGC containers. You must install Docker and the NVIDIA Container Toolkit to give containers direct GPU access:
# Install NVIDIA Container Toolkit (Ubuntu)
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# Confirm the container runtime can see your H100 or RTX 6000
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smiStep-by-Step Deployment Guide
Step A: Pull the Official TensorRT-LLM Container
NVIDIA publishes versioned TensorRT-LLM release containers on NGC. Replace x.xx.x with the current release tag (e.g., 0.10.0 or newer):
docker pull nvcr.io/nvidia/tensorrt-llm/release:x.xx.x
docker run --rm -it --ipc=host \
--ulimit memlock=-1 --ulimit stack=67108864 \
--gpus all \
-v $(pwd):/workspace \
nvcr.io/nvidia/tensorrt-llm/release:x.xx.x
# Sanity check inside the container
python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)"Using the official container resolves complex CUDA/cuDNN dependencies automatically.
Step B: Download the Model Weights
We will use Llama 3 as the working example. Ensure you have accepted the license terms on Hugging Face.
pip install -U "huggingface_hub[cli]"
huggingface-cli login # Enter your Hugging Face access token
huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct \
--local-dir /workspace/models/llama3-8bStep C: Build the TensorRT Engine (FP8 Precision)
Building an engine is a two-part process: convert the standard Hugging Face weights to TensorRT-LLM's optimized format, and then compile an engine targeting your exact GPU (H100 or RTX Pro 6000).
# 1. Convert the checkpoint and quantize to FP8
python3 examples/llama/convert_checkpoint.py \
--model_dir /workspace/models/llama3-8b \
--output_dir /workspace/checkpoints/llama3-8b-fp8 \
--dtype float16 \
--qformat fp8 \
--calib_size 1024
# 2. Compile the highly optimized TRT engine
trtllm-build \
--checkpoint_dir /workspace/checkpoints/llama3-8b-fp8 \
--output_dir /workspace/engines/llama3-8b-fp8 \
--gemm_plugin fp8 \
--max_batch_size 64 \
--max_input_len 4096 \
--max_seq_len 8192Pro-Tip: The --calib_size 1024 flag runs calibration during the FP8 conversion to prevent accuracy loss. Setting --max_batch_size appropriately based on your server's total VRAM allows TensorRT-LLM to fully utilize in-flight batching.
Step D: Serve the Model using Triton Inference Server
The compiled TensorRT-LLM engine is best served through NVIDIA's Triton Inference Server (now part of the Dynamo platform), which handles HTTP/gRPC requests securely and efficiently.
docker run --rm -it --gpus all \
--shm-size=2g \
-p 8000:8000 -p 8001:8001 -p 8002:8002 \
-v /workspace/engines/llama3-8b-fp8:/models/llama3-8b/1 \
nvcr.io/nvidia/tritonserver:xx.xx-trtllm-python-py3 \
tritonserver --model-repository=/modelsOnce Triton reports that the model is loaded, you can send API calls to port 8000 just like you would with an OpenAI-compatible endpoint.
H100 vs. RTX Pro 6000: Which GPU is Right for You?
Both GPUs are fully supported for this workflow, but they serve different business needs:
| Feature | NVIDIA H100 (SXM / PCIe) | NVIDIA RTX Pro 6000 (Ada) |
|---|---|---|
| Best Used For | Massive models (70B+), thousands of concurrent users, DeepSeek/Mixtral MoE | Smaller models (8B-35B), RAG pipelines, dev/test, AI Agent clusters |
| GPU Memory | 80 GB HBM3 | 48 GB GDDR6 w/ ECC |
| FP8 Tensor TFLOPS | ~3,958 TFLOPS | ~1,457 TFLOPS |
| Cost Profile | Premium, highest throughput | Highly cost-effective per GB of VRAM |
Conclusion & Next Steps
Deploying a Large Language Model doesn't require waiting months for unreleased hardware. The NVIDIA H100 and RTX Pro 6000 are the undisputed, proven champions of production AI today. When combined with TensorRT-LLM's FP8 precision and in-flight batching, these GPUs deliver sub-second time-to-first-token latency and massive cost savings.
FAQ
- What is TensorRT-LLM? TensorRT-LLM is NVIDIA's open-source library for optimizing and serving large language models on NVIDIA GPUs. It compiles models into highly optimized inference engines and adds LLM-specific features like in-flight batching and paged KV cache management.
- Does TensorRT-LLM support the RTX Pro 6000? Yes! The RTX Pro 6000 uses the Ada Lovelace architecture, which fully supports TensorRT-LLM's advanced features, including FP8 quantization, making it an incredibly cost-effective inference server.
- What is FP8 Quantization? FP8 is an 8-bit floating-point data format natively supported by H100 and RTX 6000 Tensor Cores. It halves the memory required for model weights compared to FP16, allowing larger models to run on fewer GPUs at doubled throughput, with virtually zero drop in response quality.
- Do I need a cluster of GPUs to run Llama 3? It depends on the model size and quantization. Using FP8 precision on TensorRT-LLM, an 8B parameter model easily fits on a single RTX Pro 6000 (48GB), leaving plenty of room for KV cache. A 70B parameter model typically requires an H100 (80GB) or a multi-GPU setup.