How to Deploy LLMs with Blackwell FP4 Precision Using TensorRT-LLM

The VRAM Bottleneck and the Blackwell Solution

Here's the math that trips up most teams planning an AI deployment: a 70B parameter model running in standard FP16 (16-bit) precision needs upwards of 140GB of VRAM just to hold the weights — before you've accounted for the KV cache, activations, or any headroom for concurrent requests. That number rules out a single GPU entirely and pushes teams straight into expensive multi-GPU clusters with NVLink interconnects, higher power draw, and a much bigger monthly bill.

NVIDIA's Blackwell architecture changes that equation. Its 5th-generation Tensor Cores add native hardware support for NVFP4, a 4-bit floating-point format that shrinks a model's memory footprint by roughly 3.5x compared to FP16 — without the accuracy collapse you'd expect from naive 4-bit quantization. That's the difference between needing a cluster and needing one card.

What Is NVFP4? A Brief Technical Primer

NVFP4 is often lumped in with older 4-bit integer quantization (INT4), but the two aren't the same thing, and the difference matters for accuracy.

Standard INT4 quantization applies a single scaling factor across a large block of weights, which tends to lose precision on outlier values — the ones that often carry the most information in a trained model. NVFP4 instead uses a two-level scaling strategy, combining fine-grained E4M3 block scaling with a broader FP32 tensor-level scale. This preserves the dynamic range of the original weights far more faithfully than flat integer quantization.

The practical result: you get the VRAM savings and throughput of 4-bit compression, but accuracy that tracks much closer to an 8-bit or even 16-bit model than traditional INT4 ever could.

Step-by-Step Tutorial: Deploying Llama 3 in FP4

The following steps take you from a bare Ubuntu server to a running FP4 inference endpoint. Each command is meant to be copied and run as-is, in order.

1. Prerequisites & Server Setup: Verify your host environment.

Before you start, confirm your server has:

• Ubuntu with the latest NVIDIA drivers installed, supporting CUDA 12.8+
• The NVIDIA Container Toolkit installed, so Docker containers can access the GPU

Verify your driver and CUDA version:

nvidia-smi

You should see your GPU listed (RTX 5090 or RTX PRO 6000) along with the CUDA version in the top-right of the output. If the NVIDIA Container Toolkit isn't installed yet, install it before moving on — TensorRT-LLM's container workflow depends on it.

2. Launch the TensorRT-LLM Environment: Avoid host dependency conflicts.

Running inside a container avoids the dependency conflicts that come from installing CUDA libraries, PyTorch, and TensorRT directly on the bare-metal host. Pull and launch the NVIDIA PyTorch container:

docker run --gpus all -it --rm \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v $(pwd):/workspace \
  nvcr.io/nvidia/pytorch:24.10-py3

This gives you a clean, GPU-enabled environment with the correct CUDA and driver compatibility already resolved, and mounts your working directory into the container.

3. Install NVIDIA ModelOpt

NVIDIA ModelOpt is the official toolkit for compressing model weights into the NVFP4 format. Install it alongside TensorRT-LLM inside your container:

pip install nvidia-modelopt tensorrt_llm -U

ModelOpt handles the calibration and quantization logic needed to convert a full-precision checkpoint into NVFP4 weights that Blackwell's Tensor Cores can execute natively.

4. Quantize the Model Weights to FP4: Python execution.

With ModelOpt installed, quantize your base model. This example uses Llama 3 8B pulled from Hugging Face. Create a Python script (quantize.py) and run it:

import modelopt.torch.quantization as mtq
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Meta-Llama-3-8B"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Apply the default NVFP4 quantization config
quant_cfg = mtq.NVFP4_DEFAULT_CFG

def calibrate(model):
    # Run a small number of forward passes with representative data
    # to calibrate the quantization scales
    sample_inputs = tokenizer("The quick brown fox", return_tensors="pt").to("cuda")
    model(**sample_inputs)

quantized_model = mtq.quantize(model, quant_cfg, forward_loop=calibrate)
quantized_model.save_pretrained("./llama3-8b-fp4")

This applies NVFP4's two-level scaling to every supported layer and saves a quantized checkpoint ready for compilation.

5. Compile the TensorRT Engine

Next, compile the quantized checkpoint into a TensorRT engine using trtllm-build:

trtllm-build \
  --checkpoint_dir ./llama3-8b-fp4 \
  --output_dir ./llama3-8b-fp4-engine \
  --use_fp4 \
  --max_batch_size 8 \
  --max_input_len 4096 \
  --max_output_len 1024

The --use_fp4 flag is the critical piece here — it instructs the compiler to target Blackwell's 5th-generation Tensor Cores directly, generating kernels that execute NVFP4 operations natively on the RTX 5090 or RTX PRO 6000 rather than falling back to a higher-precision path.

6. Run the Inference Server and Verify VRAM

Launch the compiled engine using TensorRT-LLM's standard Python runner:

python3 /workspace/TensorRT-LLM/examples/run.py \
  --engine_dir ./llama3-8b-fp4-engine \
  --tokenizer_dir "meta-llama/Meta-Llama-3-8B" \
  --max_output_len 100 \
  --input_text "Explain the benefits of dedicated GPU hosting."

With the model running, open a second terminal on the same machine and check GPU memory usage in real time:

watch -n 1 nvidia-smi

You should see VRAM consumption sitting well below what the same model would require in FP16 — this is your confirmation that the FP4 engine is running successfully.

Conclusion

Blackwell's native NVFP4 support removes one of the biggest cost barriers in self-hosted AI: VRAM. Models that used to demand multi-GPU clusters now fit comfortably on a single RTX PRO 6000 or RTX 5090, without a meaningful hit to output quality.

On top of the hardware advantage, running this stack on a bare-metal dedicated server means zero virtualization overhead — every bit of GPU compute goes toward your model, not a hypervisor.

Want to go further? Check out our guides on KV cache optimization for LLM inference and configuring Docker on Ubuntu to round out your deployment.