Training modern models is a memory problem as much as a compute problem. In 2026 the practical choice for a data-center training GPU comes down to a short list from two vendors: NVIDIA's Hopper generation (H100, H200) and Blackwell generation (B200), and AMD's Instinct line (MI300X, MI325X). Each can train large transformers, but they differ sharply in memory capacity, bandwidth, low-precision throughput, and how well they scale across many GPUs. Picking the wrong one means either paying for capability you cannot use or hitting a wall the moment your model outgrows a single accelerator.
For training specifically, four things move the needle. Memory capacity and bandwidth decide how big a model you can fit and how fast you can feed the tensor cores. Low-precision compute (FP8, BF16, and now FP4) sets peak throughput. Interconnect (NVLink and NVSwitch versus PCIe) determines whether you can scale past one GPU without the network becoming the bottleneck. And power and cooling decide what you can actually deploy in your racks. This guide walks through each, vendor-neutral, so you can match hardware to the model you intend to train.
Specs at a glance
The table below lists the mainstream data-center training GPUs. Figures are drawn from vendor datasheets and reflect the SXM/OAM (socketed) variants, which are the ones you use for serious multi-GPU training. PCIe cards exist for some parts and generally run at lower power and bandwidth.
| GPU | Memory | Bandwidth | TDP | Key precision | Architecture | Interconnect |
|---|---|---|---|---|---|---|
| NVIDIA H100 SXM | 80GB HBM3 | ~3.35 TB/s | 700W | FP8, BF16 | Hopper | NVLink 4 (~900 GB/s/GPU) |
| NVIDIA H200 SXM | 141GB HBM3e | ~4.8 TB/s | 700W | FP8, BF16 | Hopper | NVLink 4 (~900 GB/s/GPU) |
| NVIDIA B200 | 192GB HBM3e | ~8 TB/s | ~1000W | FP4, FP8, BF16 | Blackwell (dual-die) | NVLink 5 (~1.8 TB/s/GPU) |
| AMD MI300X | 192GB HBM3 | ~5.3 TB/s | 750W | FP8, BF16 | CDNA 3 | Infinity Fabric |
| AMD MI325X | 256GB HBM3e | ~6 TB/s | ~1000W | FP8, BF16 | CDNA 3 | Infinity Fabric |
A few things stand out. AMD leads on raw capacity per GPU (MI325X at 256GB), which matters for fitting large models with fewer partitions. NVIDIA's B200 leads on bandwidth (roughly 8 TB/s) and is the only part here with hardware FP4. The Hopper parts (H100, H200) remain the volume workhorses and are the most widely available, both to buy and to rent.
Memory and bandwidth
Memory is the first constraint you hit in training, and it is not just the model weights. For every parameter you also store optimizer state (for Adam, two additional values per parameter), gradients, and activations for the backward pass. A rough rule for mixed-precision Adam training is 16 to 20 bytes per parameter once weights, gradients, and optimizer moments are counted, before activations. A 70B-parameter model therefore needs well over a terabyte of aggregate GPU memory to train conventionally, which is why training always spans multiple GPUs and why per-GPU capacity determines how many you need and how you shard.
Higher capacity per GPU (the 192GB and 256GB AMD parts, or the 141GB H200) lets you hold more of the model and a larger micro-batch on each device. That reduces the degree of tensor or pipeline parallelism you need, which in turn cuts communication overhead. Bandwidth is the second half of the story: training is frequently memory-bandwidth bound, especially in attention and in the optimizer step, so the jump from H100's ~3.35 TB/s to H200's ~4.8 TB/s and B200's ~8 TB/s translates fairly directly into higher tensor-core utilization on the same model. Bigger batches also improve arithmetic intensity, letting the compute units stay busy rather than waiting on HBM.
Compute and precision (FP8, BF16, FP4)
Lower numerical precision is the single biggest lever on training throughput. BF16 has been the safe default for stable training for years. FP8 roughly doubles usable tensor-core throughput versus BF16 and, with careful scaling, can train large transformers to comparable quality. On NVIDIA hardware this is orchestrated by the Transformer Engine, which dynamically manages per-tensor scaling factors and mixes FP8 and higher-precision accumulation so that dynamic range does not collapse during training. AMD's CDNA 3 parts (MI300X, MI325X) also support FP8 in their matrix engines and are supported by ROCm's software stack for mixed-precision training.
Blackwell (B200) adds hardware FP4, a 4-bit format, along with a second-generation Transformer Engine and micro-scaling formats designed to keep 4-bit numerics usable. FP4 roughly doubles peak throughput again over FP8 for the operations that tolerate it. In practice FP4 is most mature for inference today, and its use in training is selective (applied to the layers and phases that can absorb the reduced precision) rather than blanket. Treat headline FP4 FLOPS as a ceiling, not a number you will hit across an entire training run. For most teams in 2026, BF16 and FP8 remain the precisions that carry real training workloads, with FP4 an emerging accelerator on Blackwell.
Multi-GPU scaling and interconnect
No frontier or even mid-size model trains on one GPU, so interconnect is as important as the GPU itself. During training, GPUs constantly exchange gradients (data parallelism), activations (pipeline parallelism), and shards of individual layers (tensor parallelism). Tensor parallelism in particular demands very high bandwidth and low latency between GPUs, because it communicates inside every forward and backward pass.
This is where NVLink and NVSwitch matter. Within an 8-GPU HGX node, NVSwitch gives every GPU a high-bandwidth, all-to-all path to every other GPU. Hopper's NVLink 4 provides roughly 900 GB/s of aggregate bidirectional bandwidth per GPU; Blackwell's NVLink 5 roughly doubles that to about 1.8 TB/s per GPU. PCIe, by contrast, tops out far lower and forces GPU-to-GPU traffic through the host, which throttles tensor-parallel training. AMD uses Infinity Fabric to link the eight GPUs on an OAM baseboard with a similar all-to-all intent. The standard building block on both sides is the 8-GPU node (NVIDIA HGX, AMD Instinct platform); you then scale out across nodes with InfiniBand or high-speed Ethernet. The takeaway for buyers: if you plan to use tensor parallelism or train models that span GPUs, socketed NVLink/Infinity-Fabric nodes beat PCIe cards, and the gap widens as you add GPUs.
Real-world training performance
Peak FLOPS rarely survive contact with a real training run, so lean on standardized and vendor-reported results, and read the fine print. In MLPerf Training submissions, NVIDIA has reported the B200 delivering up to roughly 2.2x the per-GPU training performance of the H100 on comparable workloads. NVIDIA has separately claimed up to 3x faster training for Blackwell versus the previous generation in larger, system-level configurations that lean on FP4/FP8, NVLink 5, and bigger NVSwitch domains, along with nearly 2x better training performance per dollar in those setups. The higher figure reflects full-system and lower-precision advantages, not a like-for-like single-GPU comparison, so expect the real gain on your workload to land somewhere in that range depending on model, precision, and parallelism strategy.
For AMD, independent training benchmarks are thinner than NVIDIA's, and much of the public MI300X/MI325X performance data centers on inference, where the large memory shines. For training, the honest position is that CDNA 3 parts are competitive on memory-bound and capacity-bound workloads, while ROCm software maturity and framework coverage are the variables to validate against your own stack. Whatever the source, benchmark your actual model before committing at scale: tokenizer, sequence length, parallelism, and precision choices swing results more than any headline multiplier.
Power, cooling, and deployment
Power has become a first-order deployment constraint. The Hopper and MI300X parts sit around 700 to 750W per GPU, which air-cooled 8-GPU nodes can generally handle, though a fully loaded HGX H100/H200 node still draws well over 10 kW once CPUs, memory, NICs, and fans are counted. The newer high-end parts (B200 and MI325X) push to roughly 1000W per GPU, and dense Blackwell configurations increasingly assume liquid cooling. That changes your facility requirements: rack power density, PDU sizing, and whether your data center supports direct-to-chip liquid loops rather than air alone.
Before you order, plan the rack, not just the card. A node that draws 10 to 14 kW changes how many you fit per rack and what cooling you need. Model the numbers with the GPU Vendors Power Calculator to size per-node and per-rack draw, and the Cooling Calculator to check whether air cooling is viable or liquid is required at your target density. Getting this wrong is a common reason otherwise-good GPU purchases sit underutilized.
Price, value, and rent vs buy
Prices move too fast to quote responsibly, but the tiers are stable. Hopper (H100, H200) and MI300X occupy the mainstream tier and have the deepest supply, which makes them the value picks for most training. B200 and MI325X sit at the top tier, commanding a premium for their bandwidth, capacity, and (on Blackwell) FP4. As a rough guide to value, weigh total cost per useful training-hour, which folds in throughput, power, and utilization, rather than sticker price per GPU.
The rent-versus-buy decision hinges on utilization. If you will keep GPUs busy above roughly 60 to 70 percent around the clock for a year or more, buying (or a long reserved commitment) usually wins on total cost. If your training is bursty (periodic fine-tunes, experiments, or a single large run), renting avoids capital outlay and the risk of owning depreciating hardware through a fast product cycle. Renting also lets you train on Blackwell today without a facility upgrade. Run your own numbers with the GPU Vendors Rent vs Buy Calculator using your real duty cycle and time horizon.
Recommendations by use case
| Profile | Recommended | Why |
|---|---|---|
| Startup / single-node | H100 or H200 (8-GPU HGX), or rent | Best availability and price-to-performance; H200's 141GB eases larger models. |
| Scaling team | H200 or B200 nodes; MI325X if capacity-bound | Balance throughput, memory, and multi-node NVLink/InfiniBand scaling. |
| Frontier lab | B200 at scale | Highest bandwidth, FP4, NVLink 5, best system-level scaling. |
| Capacity-first / cost-sensitive | AMD MI300X / MI325X | Largest memory per GPU; validate ROCm coverage for your framework. |
On NVIDIA versus AMD: NVIDIA's advantage is software maturity (CUDA, cuDNN, Transformer Engine) and the broadest ecosystem and support. AMD's advantage is memory capacity per GPU and often favorable pricing, with the caveat that you should validate ROCm support for your exact training stack (frameworks, custom kernels, distributed libraries) before committing.
Compare live prices from verified vendors
- NVIDIA B200 192GB
- NVIDIA H200 141GB
- NVIDIA H100 80GB
- AMD Instinct MI300X 192GB
- AMD Instinct MI325X 256GB
- Browse AI GPU servers or use the Server Configurator.
FAQ
How much VRAM do I need to train a 70B model? More than a single GPU holds. With mixed-precision Adam you should budget roughly 16 to 20 bytes per parameter for weights, gradients, and optimizer state, plus activations, which puts a 70B model well over 1TB of aggregate GPU memory. In practice that means a full 8-GPU node (or more) using sharding techniques like FSDP or ZeRO, or tensor and pipeline parallelism. Higher-capacity GPUs (141GB to 256GB) reduce how aggressively you have to shard.
NVIDIA or AMD for training? NVIDIA is the lower-risk default because of CUDA and ecosystem maturity, and it has the widest availability. AMD (MI300X, MI325X) offers more memory per GPU and competitive pricing, and is a strong option if your framework and kernels are well supported on ROCm. Validate your specific training stack on AMD before scaling.
Do I need NVLink to train? For multi-GPU training, yes, effectively. NVLink and NVSwitch (or AMD Infinity Fabric) provide the GPU-to-GPU bandwidth that tensor and pipeline parallelism depend on. PCIe-only setups work for small-scale or data-parallel jobs but become a bottleneck as you add GPUs or use tensor parallelism.
Is renting cheaper than buying? It depends on utilization. For bursty or short-term training, renting is usually cheaper and avoids capital risk. For sustained high-utilization workloads over a year or more, buying or a reserved commitment typically wins. Model it with the Rent vs Buy Calculator.
What is FP4 and should I train with it? FP4 is a 4-bit format introduced in hardware on Blackwell (B200) that roughly doubles peak throughput over FP8. It is most mature for inference today and is applied selectively in training rather than across an entire run. BF16 and FP8 remain the reliable training precisions for most teams.
Air or liquid cooling? Around 700 to 750W per GPU, air-cooled nodes are generally viable. At roughly 1000W per GPU (B200, MI325X) and high rack density, liquid cooling is increasingly required. Check your target density with the Cooling Calculator before ordering.