Choosing a GPU for AI inference is a different problem from choosing one for training. Training rewards raw compute and multi-GPU scaling. Inference, and especially large language model (LLM) serving, is dominated by memory and memory bandwidth. If you are deploying inference at scale, five factors decide whether your deployment is fast, cheap, and reliable:
- Memory capacity for weights plus KV-cache. The model weights have to fit, and so does the key/value cache that grows with every concurrent request and every token of context. Run out of memory and you either shard across more GPUs or drop concurrency.
- Memory bandwidth for tokens per second. During autoregressive decoding, each new token requires reading the full set of active weights from memory. Decode throughput tracks bandwidth far more closely than it tracks peak FLOPS.
- Low-precision support (INT8, FP8, FP4). Quantization shrinks the memory footprint and increases effective throughput. Newer architectures add native FP8 and FP4 paths that materially change cost per token.
- Latency versus throughput. Interactive chat wants low time-to-first-token and fast per-token latency. Batch and offline workloads want maximum aggregate tokens per second. The right GPU depends on which you are optimizing.
- Cost per token. The only metric that ties it all together. A more expensive GPU that serves many more tokens per second can be cheaper per token than a budget card.
This guide is vendor-neutral. It compares the GPUs most teams actually shortlist for inference in 2026 and gives you a framework for matching hardware to workload.
Specs at a glance
All figures below are drawn from vendor datasheets and reputable spec references. Bandwidth and TDP vary slightly by board partner and form factor, so treat single numbers as roughly representative.
| GPU | Memory | Bandwidth | TDP | Key precision | Best for |
|---|---|---|---|---|---|
| NVIDIA L4 | 24 GB GDDR6 | ~300 GB/s | 72 W | INT8, FP8 (Ada) | Edge, small models, high-density racks |
| NVIDIA A10 | 24 GB GDDR6 | ~600 GB/s | 150 W | INT8, FP16 (Ampere) | Budget mainstream, mixed AI plus graphics |
| NVIDIA L40S | 48 GB GDDR6 | ~864 GB/s | 350 W | FP8, INT8 (Ada) | Mainstream serving, mid-size models |
| NVIDIA H200 | 141 GB HBM3e | ~4.8 TB/s | 700 W | FP8 (Hopper) | Large-model serving, long context |
| NVIDIA B200 | 192 GB HBM3e | ~8 TB/s | ~1000 W | FP4, FP8 (Blackwell) | Frontier serving, highest throughput |
| AMD Instinct MI300X | 192 GB HBM3 | ~5.3 TB/s | 750 W | FP8, FP16 (CDNA 3) | Large-model serving, memory-heavy workloads |
A few things jump out. The gap in memory bandwidth between the GDDR6 cards (hundreds of GB/s) and the HBM cards (multiple TB/s) is close to an order of magnitude, and that gap is what separates edge-class from frontier-class serving. The gap in memory capacity, from 24 GB to 192 GB, is what determines how many GPUs you need to hold a given model.
Memory, KV-cache, and context length
Three things consume GPU memory during inference: the model weights, the KV-cache, and activation and runtime overhead.
Weights are the fixed cost. A rough rule is 2 bytes per parameter at FP16/BF16, 1 byte at FP8/INT8, and about half a byte at 4-bit. So a 70-billion-parameter model needs roughly 140 GB at FP16, roughly 70 GB at 8-bit, and roughly 35 to 40 GB at 4-bit. That single fact explains why a 192 GB GPU can host a 70B model comfortably on one card while a 24 GB GPU cannot host it at all without aggressive quantization and sharding.
KV-cache is the variable cost, and it is the part teams underestimate. Every token in every active request stores a key and value vector in each layer. KV-cache size scales with the number of concurrent requests, the sequence length (prompt plus generated tokens), the number of layers, and the hidden dimension. For large models with long context windows and high concurrency, the KV-cache can rival or exceed the weights in size. Doubling your context window or doubling batch size can double KV-cache memory.
This is why big-memory GPUs reduce GPU count. If weights plus KV-cache fit on one card, you avoid tensor-parallel sharding across two, four, or eight GPUs. Fewer GPUs means less cross-GPU communication, simpler deployment, better utilization, and often lower cost per token. A 141 GB H200 or a 192 GB B200 or MI300X can serve a large model and a healthy concurrency budget from a single device where an 80 GB-class card would force sharding.
Bandwidth and tokens per second
LLM decoding is memory-bandwidth-bound. In the decode phase, the model generates one token at a time, and each step streams the active weights from GPU memory through the compute units. With a small batch, arithmetic intensity is low, so the GPU spends most of its time waiting on memory, not computing. The practical ceiling on single-stream tokens per second is roughly memory bandwidth divided by the bytes of weights read per token.
That relationship is why the HBM cards dominate large-model serving. The H200 at roughly 4.8 TB/s, the MI300X at roughly 5.3 TB/s, and the B200 at roughly 8 TB/s move data many times faster than GDDR6 cards in the hundreds of GB/s. For a large model, that bandwidth advantage translates almost directly into higher per-user token rates and higher aggregate throughput.
The prefill phase (processing the input prompt) behaves differently. Prefill is compute-heavy and parallel, so it leans on raw FLOPS and benefits from the tensor-core throughput of the high-end parts. A well-tuned serving stack overlaps prefill and decode across requests, so both bandwidth and compute matter, but for the token-by-token generation your users feel, bandwidth is king.
Precision: INT8, FP8, FP4 and quantization
Quantization is the single highest-leverage lever for inference cost. Serving a model in lower precision reduces the memory the weights occupy and the bytes read per token, which cuts both the GPU count and the per-token latency.
- INT8 and FP8 roughly halve memory versus FP16 and are widely supported. FP8 in particular (native on Hopper, Ada L40S, Blackwell, and AMD CDNA 3) has become the default production format for many LLM deployments because it preserves accuracy well while cutting footprint and boosting throughput.
- FP4 is the newer frontier. NVIDIA's Blackwell architecture (B200) adds native FP4 tensor-core support, roughly halving memory again versus FP8 and pushing peak low-precision throughput far higher. For very large models served at high volume, FP4 can be the difference between fitting on one GPU or two.
The tradeoff is accuracy. Lower precision introduces quantization error. In practice, 8-bit inference is close to lossless for most models with modern calibration and per-channel or per-tensor scaling. Four-bit weight quantization is usually acceptable for many workloads but can degrade quality on reasoning-heavy or long-tail tasks, so it should be validated against your own evals rather than assumed. The safe pattern is to quantize, measure on representative prompts, and only ship the precision that holds your quality bar.
Latency versus throughput (and cost per token)
Batching is the core tension in inference economics. Serving one request at a time gives the lowest latency but wastes the GPU, because decode is memory-bound and a single stream cannot saturate the hardware. Increasing batch size and concurrency raises aggregate tokens per second (better throughput and lower cost per token) but adds queueing and can raise per-request latency.
A useful way to think about cost per token, without inventing prices:
Cost per token ≈ (GPU cost per hour) ÷ (tokens per second at your target latency × 3600)
Two levers move that denominator. First, pick a GPU whose bandwidth and memory let you run a large enough batch at your latency target. Second, quantize so each token reads fewer bytes. The GPU cost per hour is the numerator, and this is exactly why a pricier high-throughput card can win: if a B200 or MI300X serves several times the tokens per second of a smaller card, its higher hourly cost is spread across far more tokens.
The decision framework:
- Latency-critical, low or spiky volume, small model. A smaller GPU (L4, A10, L40S) that comfortably holds the model often gives the best cost per token, because you are not paying for bandwidth you cannot fill.
- High, steady volume, large model. A big-memory, high-bandwidth GPU (H200, B200, MI300X) run at high batch size almost always wins on cost per token, even though the sticker price per hour is higher.
Tiers: budget, mainstream, high-throughput
Budget and edge (L4, A10). The L4 is a 72 W, single-slot card built for density and efficiency. It is ideal for edge deployments, smaller models, embeddings, vision, and high-density racks where power and cooling are the constraint. The A10 offers more bandwidth (~600 GB/s) at 150 W and is a solid budget mainstream option for small to mid-size models and mixed AI-plus-graphics workloads. Neither has the memory to host large LLMs without heavy quantization.
Mainstream (L40S). With 48 GB of GDDR6, ~864 GB/s of bandwidth, and native FP8, the L40S is the workhorse for mid-size models and general API serving. It hosts many popular open models (in the roughly 7B to 34B range, and larger with quantization) on a single card and pairs well with moderate concurrency. It is the natural step up when the L4/A10 run out of memory or throughput.
High-throughput and frontier (H200, B200, MI300X). These are the cards for serving large models at scale. The H200's 141 GB of HBM3e and 4.8 TB/s handle 70B-class models and long context on one GPU. The MI300X pushes memory to 192 GB and bandwidth to ~5.3 TB/s, a strong option for memory-heavy serving and a credible non-NVIDIA path as the ROCm software stack has matured. The B200 sits at the top with 192 GB, ~8 TB/s, and native FP4, delivering the highest throughput and the best cost per token at very high volume, at the price of ~1000 W per GPU and demanding infrastructure.
Power and deployment
Power and cooling are not afterthoughts at these TDPs. An L4 sips 72 W, but a rack of B200s at roughly 1000 W each imposes serious demands on power delivery, cooling (frequently liquid at the high end), and rack density. Before you commit to a configuration, model the full-system draw, not just the GPU. Account for CPUs, memory, networking, and cooling overhead, and confirm your facility or colocation provider can supply and cool it.
Estimate total draw for any configuration with the GPU Vendors power calculator before you buy.
Rent vs buy for inference
The rent-versus-buy decision comes down to load shape and utilization.
- Steady, predictable, high utilization. If you keep GPUs busy most of the day every day, owning (or a long-term reservation) usually beats on-demand rental on total cost. Inference at scale often fits this profile because production traffic is continuous.
- Bursty, spiky, or uncertain. If your load swings widely, is seasonal, or you are still validating product-market fit, renting avoids stranded capital and lets you scale elastically. Paying a premium per hour is worth it when you would otherwise pay for idle hardware.
Many teams land on a hybrid: own or reserve a baseline sized to steady traffic, and burst to rented capacity for peaks. Model both paths for your utilization with the rent-vs-buy calculator.
Recommendations by use case
| Use case | Recommended GPUs | Why |
|---|---|---|
| Edge and small models | NVIDIA L4, A10 | Low power, high density, enough memory for small models, embeddings, and vision |
| Mainstream API serving | NVIDIA L40S | 48 GB and FP8 host mid-size models on one card with good throughput per dollar |
| Large-model serving | NVIDIA H200, AMD MI300X | 141 to 192 GB fits 70B-class models plus KV-cache on a single GPU |
| Frontier, high-volume | NVIDIA B200, AMD MI300X | Highest bandwidth and (Blackwell) FP4 for the best cost per token at scale |
| Long-context workloads | H200, B200, MI300X | Large HBM absorbs big KV-caches without extra sharding |
Compare live prices from verified vendors
FAQ
How much VRAM do I need to serve a 70B model? At FP16 the weights alone are roughly 140 GB, so you need multiple GPUs or a big-memory card. At 8-bit the weights drop to roughly 70 GB, which fits comfortably on a 141 GB H200 or a 192 GB B200 or MI300X with room left for KV-cache and concurrency. At 4-bit you can get the weights under 40 GB, but you still need headroom for the KV-cache at your target context length and batch size. For production, plan for weights plus a generous KV-cache budget, not just the weights.
Is FP8 accurate enough for production? For most models, yes. Modern FP8 quantization with proper calibration is close to lossless in practice and has become a default production format. Always validate against your own evaluation set, and reserve FP16 for the rare quality-critical path that measurably regresses under FP8.
What is the best budget GPU for inference? For edge and small models, the L4 is hard to beat on power efficiency and rack density at 72 W. If you need more bandwidth for slightly larger models, the A10 at 150 W is a strong budget step up. Move to the L40S once memory or throughput becomes the bottleneck.
When does FP4 make sense? FP4, native on Blackwell (B200), makes sense for very large models served at high volume, where halving memory again versus FP8 lets you fit on fewer GPUs and push higher throughput. Validate quality carefully, since 4-bit can degrade accuracy on harder tasks.
Should I use the cloud or own hardware for inference? Own or reserve when utilization is high and steady, since continuous production traffic amortizes the capital well. Rent when load is bursty, seasonal, or uncertain. A hybrid (owned baseline plus rented burst capacity) fits many teams. Model both with the rent-vs-buy calculator linked above.
Does AMD MI300X compete with NVIDIA for inference? Yes, particularly for memory-heavy and large-model serving. With 192 GB of HBM3 and roughly 5.3 TB/s of bandwidth, the MI300X can host large models on a single GPU, and the ROCm software stack has matured to make it a credible option. Validate your specific model and serving framework support before committing.