NVIDIA · Volta

What runs on a V100 32GB

32 GB at 900 GB/s. The largest model that fits at Q4_K_M is Mixtral 8x7B, at roughly 58.5 tokens per second.

Runs comfortably

6.4 GB of 29.3 GB · 22%
029 GB
Weights 4.6 GB
KV cache 1.0 GB
Runtime overhead 0.8 GB

Llama 3.1 8B at Q4_K_M leaves 22.8 GB spare on a V100 32GB. There is room to raise the context length or move up a quantisation level.

Generation106tok/s
Prompt processing3269tok/s
Max context128Ktokens
KV per 1K tokens0GB

Every quantisation of Llama 3.1 8B on a V100 32GB

Highlighted row is the highest quality that still fits at 8K context.

QuantisationbpwWeights TotalVerdictMax ctxtok/s
FP16 / BF16 16.00 15.0 GB 16.8 GB Runs comfortably 108K 35.4
INT8 / W8A8 8.50 7.7 GB 9.5 GB Runs comfortably 128K 66.3
Q8_0 (GGUF) 8.50 7.7 GB 9.5 GB Runs comfortably 128K 66.3
FP8 (E4M3) 8.00 7.5 GB 9.3 GB Runs comfortably 128K 68.2
Q6_K 6.56 6.1 GB 8.0 GB Runs comfortably 128K 81.9
AWQ 4-bit 4.25 5.4 GB 7.2 GB Runs comfortably 128K 91.7
GPTQ 4-bit 4.25 5.4 GB 7.2 GB Runs comfortably 128K 91.7
MXFP4 4.25 5.4 GB 7.2 GB Runs comfortably 128K 91.7
Q5_K_M 5.67 5.3 GB 7.1 GB Runs comfortably 128K 93.4
Q5_K_S 5.52 5.2 GB 7.0 GB Runs comfortably 128K 95.7
Q4_K_M 4.85 4.6 GB 6.4 GB Runs comfortably 128K 106
Q4_K_S 4.58 4.4 GB 6.2 GB Runs comfortably 128K 110
Q4_0 4.55 4.4 GB 6.2 GB Runs comfortably 128K 111
IQ4_XS 4.25 4.1 GB 5.9 GB Runs comfortably 128K 117
Q3_K_M 3.91 3.8 GB 5.7 GB Runs comfortably 128K 124
IQ3_M 3.70 3.7 GB 5.5 GB Runs comfortably 128K 129
IQ3_XXS 3.06 3.2 GB 5.0 GB Runs comfortably 128K 146
Q2_K 2.63 2.8 GB 4.6 GB Runs comfortably 128K 161
IQ2_XXS 2.06 2.3 GB 4.2 GB Runs comfortably 128K 187
IQ1_M 1.75 2.1 GB 3.9 GB Runs comfortably 128K 204

Models on a V100 32GB

Q4_K_M at 8K context. Bandwidth sets the speed; capacity sets the ceiling.

ModelParamsWeightsVerdictMax ctxtok/s
DeepSeek-R1 671B-A37B 671B 379.0 GB Won't fit — 1.95
Qwen3 235B-A22B 235.1B 132.8 GB Won't fit — 3.52
gpt-oss 120B-A5.1B 116.8B 66.0 GB Won't fit — 19.4
Llama 4 Scout 109B-A17B 109B 61.7 GB Won't fit — 6.09
Qwen2.5 72B 72.7B 41.2 GB Won't fit — 2.19
Llama 3.1 70B 70.5B 40.0 GB Won't fit — 2.32
Mixtral 8x7B 46.7B 26.4 GB Fits, but tight 17K 58.5
Command R 35B 35.0B 20.1 GB Won't fit 7K 9.50
Yi-1.5 34B 34.4B 19.5 GB Runs comfortably 32K 26.7
Qwen3 32B 32.8B 18.6 GB Runs comfortably 39K 27.8
Qwen2.5-Coder 32B 32.8B 18.6 GB Runs comfortably 39K 27.8
DeepSeek-R1-Distill-Qwen 32B 32.8B 18.6 GB Runs comfortably 39K 27.8
Qwen3 30B-A3B 30.5B 17.3 GB Runs comfortably 120K 113
Gemma 3 27B 27.4B 15.6 GB Runs comfortably 128K 33.4
Mistral Small 24B 23.6B 13.4 GB Runs comfortably 32K 38.9
gpt-oss 20B-A3.6B 20.9B 11.9 GB Runs comfortably 128K 174
Qwen3 14B 14.8B 8.5 GB Runs comfortably 128K 59.9
Phi-4 14B 14.7B 8.4 GB Runs comfortably 16K 59.4
Mistral NeMo 12B 12.3B 7.0 GB Runs comfortably 128K 70.9
Gemma 3 12B 12.2B 7.0 GB Runs comfortably 128K 71.8
GLM-4 9B 9.4B 5.4 GB Runs comfortably 128K 96.9
Qwen3 8B 8.2B 4.7 GB Runs comfortably 128K 102
Llama 3.1 8B 8.0B 4.6 GB Runs comfortably 128K 106
DeepSeek-R1-Distill-Llama 8B 8.0B 4.6 GB Runs comfortably 128K 106
Qwen2.5 7B 7.6B 4.4 GB Runs comfortably 128K 117
Mistral 7B v0.3 7.3B 4.1 GB Runs comfortably 32K 117
Gemma 3 4B 4.3B 2.5 GB Runs comfortably 128K 197
Qwen3 4B 4.0B 2.3 GB Runs comfortably 128K 185
Llama 3.2 3B 3.2B 1.8 GB Runs comfortably 128K 232
Llama 3.2 1B 1.2B 0.7 GB Runs comfortably 128K 614

Direct answers

Llama 3.1 8B on V100 32GB
8.0BLlama 3.1 70B on V100 32GB
70.5BLlama 3.2 3B on V100 32GB
3.2BLlama 3.2 1B on V100 32GB
1.2BLlama 4 Scout 109B-A17B on V100 32GB
109BQwen3 8B on V100 32GB
8.2BQwen3 14B on V100 32GB
14.8BQwen3 32B on V100 32GB
32.8B