Multi-Instance GPU (MIG) Partitioning on A100/H100: Run Multiple AI Models on One Card

Key Takeaways

• MIG is hardware-level partitioning, not time-slicing. Each MIG instance gets a dedicated slice of Streaming Multiprocessors (SMs), memory, and L2 cache, so workloads on separate instances cannot interfere with each other.
• A100 supports up to 7 instances (profiles from 1g.5gb to 7g.40gb on the 40GB card); H100 also supports up to 7 instances (profiles from 1g.10gb to 7g.80gb on the 80GB card).
• Setup is CLI-driven: enable MIG mode with nvidia-smi -mig 1, then create GPU Instances and Compute Instances with nvidia-smi mig -cgi and -C.
• MIG instances are not persistent across reboots by default — plan for automated reconfiguration in production using NVIDIA's mig-parted tool.

What is Multi-Instance GPU (MIG)?

Multi-Instance GPU (MIG) is a hardware capability built into NVIDIA's Ampere, Hopper, and newer datacenter GPUs. It splits one physical GPU into multiple fully isolated GPU Instances, each behaving like its own standalone CUDA device.

Every instance owns a dedicated fraction of the GPU's Streaming Multiprocessors (SMs), a fixed slice of high-bandwidth memory, and a portion of the L2 cache and memory controllers. A workload running on one instance cannot see, starve, or slow down a workload running on another — this is the core difference between MIG and older GPU-sharing approaches.

MIG Profiles: A100 vs. H100

NVIDIA ships a fixed set of "profiles" per GPU model. A profile defines how much compute (in SM slices) and memory an instance gets. Profile names follow the pattern <compute slices>g.<memory>gb — for example, 3g.20gb means 3/7 of the SMs and 20GB of memory.

Uniform vs. Mixed Geometry

You can partition a GPU two ways. A uniform geometry creates several identical instances — for example, seven 1g.5gb slices on an A100 for seven small inference endpoints. A mixed geometry combines different profile sizes on the same card, such as one 3g.40gb instance for a mid-size model plus four 1g.10gb instances for lightweight services.

Mixed geometry is where MIG earns its keep for AI hosting: you right-size each instance to the model it's actually running instead of forcing every workload onto identical, oversized slices.

Prerequisites Before You Start

• Supported GPU: A100, A30, H100, H200, or newer (MIG is not supported on consumer RTX cards used outside the RTX PRO Blackwell line).
• Driver and CUDA: A100/A30 workloads need CUDA 11.0+; H100/H200 workloads need CUDA 12.0+. Always run the latest NVIDIA datacenter driver, since specific profiles (like A100's 1g.10gb) are gated behind minimum driver versions.
• Root/administrative access to run nvidia-smi mig commands, or read access to the mig/config capability if a non-root user will manage instances.
• NVIDIA Container Toolkit v2.5.0+ if you plan to run workloads inside Docker containers — see our NVIDIA GPU passthrough on Ubuntu 24.04 guide if you're setting this up on bare metal for the first time.

Step-by-Step: Partitioning a GPU with MIG

Step 1: Check Current MIG Status

nvidia-smi -i 0

Look at the bottom-right of the output. A fresh A100 or H100 will show MIG M.: Disabled.

Step 2: Enable MIG Mode

sudo nvidia-smi -i 0 -mig 1

On Ampere GPUs (A100/A30), the driver resets the GPU to apply MIG mode, and the setting persists across reboots until explicitly disabled. On Hopper and newer GPUs (H100/H200), no reset is required, but MIG mode does not persist across a driver reload — you'll need to re-enable it if the kernel modules unload.

Step 3: List Available Profiles

nvidia-smi mig -lgip

This prints every profile the card supports, including its ID, free/total instance count, and memory size. Use it to confirm exact numbers for your specific GPU and driver version before planning a geometry.

Step 4: Create GPU Instances and Compute Instances

Use -cgi to create GPU Instances (GIs) and -C to auto-create the matching Compute Instances (CIs) in the same step. You can reference a profile by ID or by short name.

# Example: split one A100 into two equal 3g.20gb instances
sudo nvidia-smi mig -cgi 9,3g.20gb -C

# Example: mixed geometry — one 4g.20gb instance plus one 2g.10gb and one 1g.5gb
sudo nvidia-smi mig -cgi 5,14,19 -C

Step 5: Verify the Instances

nvidia-smi mig -lgi
nvidia-smi -L

The second command lists each MIG device's UUID (e.g., MIG-c7384736-a75d-5afc-978f-d2f1294409fd) — you'll need these to target a specific instance when launching a workload.

Running Multiple Models Concurrently

Once instances exist, pin each workload to its own MIG device using CUDA_VISIBLE_DEVICES. This is what actually lets you run two or more models on one physical card at the same time.

Bare Metal

CUDA_VISIBLE_DEVICES=MIG-c7384736-a75d-5afc-978f-d2f1294409fd python serve_model_a.py &
CUDA_VISIBLE_DEVICES=MIG-a28ad590-3fda-56dd-84fc-0a0b96edc58d python serve_model_b.py &

Docker

docker run --gpus '"device=0:0"' \
  nvcr.io/nvidia/pytorch:24.xx-py3 \
  python /app/serve_model_a.py

The device=0:0 syntax targets GPU index 0, MIG device index 0. Each container only sees the single MIG instance you assign it; the rest of the physical GPU is invisible to it.

A Practical Example: On a single H100 80GB, you could carve out one 3g.40gb instance to serve a 13B-parameter model with room for KV cache, and use the remaining capacity as two 2g.20gb instances for smaller embedding or reranking models. That’s three independent services, one GPU, and no resource contention between them.

Reconfiguring and Production Best Practices

• MIG geometry is dynamic but not persistent. Instances don't survive a system reboot by default. For production, use NVIDIA's MIG Partition Editor (mig-parted) to define and reapply a geometry automatically via a systemd service.
• The GPU must be idle to reconfigure. You can't resize a running instance. Destroy it with nvidia-smi mig -dci and -dgi, then recreate the new geometry.
• Shift geometry to match demand patterns. A common pattern is running seven small inference instances during business hours, then tearing them down for one large 7g instance to run an overnight training or fine-tuning job.
• Use DCGM for monitoring on Ampere GPUs. On A100/A30, standard nvidia-smi utilization metrics show as N/A per MIG instance. NVIDIA DCGM v2.0.13+ is needed for per-instance profiling data.

When MIG Is the Wrong Tool

MIG isn't universal. Skip it if a single workload needs the GPU's full memory bandwidth or all of its SMs — like training a large model that already saturates a full A100 or H100. It's also not the right fit when a job needs more memory than any single instance profile provides, since instances can't be combined after creation without tearing down and rebuilding the whole geometry.

FAQ: MIG Partitioning on A100/H100

• Does MIG work on RTX or consumer GPUs? No. MIG is supported on NVIDIA's datacenter-class GPUs (A100, A30, H100, H200, B200) and on the RTX PRO 6000/5000/4500 Blackwell workstation and server editions. Standard GeForce RTX consumer cards do not support MIG.
• What's the difference between MIG and vGPU? MIG creates hardware-isolated GPU instances at the driver level for bare-metal or containerized CUDA workloads, with each instance behaving like an independent physical GPU. NVIDIA vGPU is a virtualization technology that creates virtual GPU devices for VMs, typically through a hypervisor, and can be layered on top of MIG instances in virtualized environments.
• Can I resize a MIG instance without downtime? No. MIG instances have a fixed size once created. To change a geometry, you must destroy the existing GPU Instances and Compute Instances with nvidia-smi mig -dci and -dgi, then create a new configuration — this briefly takes the GPU's MIG instances offline.
• Is there a performance penalty on the smallest MIG slice? A smaller instance simply has fewer SMs and less memory bandwidth than a larger one; it isn't "throttled," it's proportionally sized. A 1g.5gb instance on an A100 will run a compatible workload slower than a 7g.40gb instance, but at consistent, predictable throughput with zero interference from other instances on the card.
• Do MIG instances persist after a server reboot? No, by default. Created GPU Instances and Compute Instances are not saved across a system reboot or driver reload. For production deployments, automate geometry recreation at boot using NVIDIA's mig-parted tool.
• How many models can I run on one A100 or H100 with MIG? Up to seven, using the smallest profile size on either GPU (1g.5gb on A100, 1g.10gb on H100). The practical number depends on how much memory and compute each model actually needs — you can also mix instance sizes to match your specific model lineup.