The NVIDIA H200 and B200 are the two accelerators most AI teams are weighing right now. The H200 is the memory-boosted refresh of the proven Hopper H100. The B200 is the flagship of the newer Blackwell generation. They are not small steps apart (the B200 roughly doubles raw throughput and adds native FP4), but the H200 is cheaper, cooler, and shipping in volume today. Which one belongs in your next cluster depends less on the headline FLOPS number and more on your workload mix, your cooling envelope, and how you plan to pay for the hardware.
This guide breaks down the real differences, with verified numbers, and ends with a buyer-focused verdict.
Spec table: H200 vs B200
| Spec | NVIDIA H200 (SXM) | NVIDIA B200 (SXM) |
|---|---|---|
| Architecture | Hopper (single GH100 die) | Blackwell (dual-die, one logical GPU) |
| Memory | 141 GB HBM3e | 192 GB HBM3e |
| Memory bandwidth | 4.8 TB/s | 8.0 TB/s |
| Streaming multiprocessors | 132 | 148 (aggregate) |
| FP4 dense | Not supported | roughly 9,000 TFLOPS (9 PFLOPS) |
| FP8 dense | roughly 1,979 TFLOPS (2 PFLOPS) | roughly 4,500 TFLOPS (4.5 PFLOPS) |
| BF16/FP16 dense | roughly 989 TFLOPS | roughly 2,250 TFLOPS |
| FP64 | roughly 34 TFLOPS | roughly 37 TFLOPS |
| NVLink per GPU | 900 GB/s | 1.8 TB/s |
| TDP | 700 W (600 W on the NVL card) | roughly 1,000 W |
Figures above are dense (non-sparse) tensor throughput and reflect vendor datasheets and independent spec sheets. Sparse (2:4 structured) numbers are roughly double the dense values on both parts. Treat single-GPU peak FLOPS as a ceiling, not a promise. Real applications rarely touch it.
Product pages for reference: the NVIDIA B200 192GB and the NVIDIA H200 NVL 141GB.
Memory: 141 vs 192 GB, and why bandwidth drives token generation
The most underappreciated spec on this table is memory bandwidth, because it, not FLOPS, sets the ceiling for autoregressive token generation.
Here is the intuition. During the decode phase of LLM inference (generating one token at a time), the GPU must read the entire set of model weights from HBM for every token it produces. Decode is almost never compute-bound. It is memory-bound. So the practical upper bound on single-stream generation speed is roughly:
tokens per second ≈ memory bandwidth / bytes read per token
Take a 70B-parameter model served in FP8, where each parameter is one byte, so roughly 70 GB must move per token. On the H200 at 4.8 TB/s, the ceiling is about 4,800 / 70, or roughly 68 tokens per second per GPU for a single stream. On the B200 at 8.0 TB/s, it is about 8,000 / 70, or roughly 114 tokens per second. That is a 1.6x to 1.7x gap driven purely by bandwidth, before any compute or FP4 advantage enters the picture. Serve the same model in BF16 (two bytes per parameter) and both ceilings roughly halve, which is exactly why quantization matters so much for interactive latency.
Capacity matters too, and in a step-function way. The extra 51 GB on the B200 (192 vs 141) decides whether a model plus its KV cache fits on one GPU or spills across two. A 70B model in FP8 weights (about 70 GB) leaves the H200 roughly 60 GB for KV cache and activations, which is fine for moderate context and batch sizes but tightens quickly at long context. The B200's 192 GB gives meaningfully more room for large batches, long context windows, or fitting a bigger model without tensor-parallel splitting. Avoiding that split is not just a capacity win. Every cross-GPU shard adds NVLink traffic and synchronization overhead, so consolidating onto fewer, larger GPUs often improves throughput per dollar and simplifies deployment.
Both parts use HBM3e. The B200 gets its bandwidth edge from a wider effective interface across its dual-die package rather than from radically faster memory chips.
You can model your own memory headroom and batch sizing in the server configurator.
Compute: FP8, the new FP4, and where roughly 2x comes from
On compute, the Blackwell B200 is a genuine generational jump, not a refresh.
At FP8, the format most production inference and a lot of training now use, the B200 delivers roughly 4.5 PFLOPS dense against the H200's roughly 2 PFLOPS. That is about 2.25x. At BF16, the numbers are roughly 2,250 vs 989 TFLOPS, again a bit over 2x. This is the origin of the widely quoted "B200 is about twice an H200" claim. For dense throughput at matched precision, that holds up: expect roughly 2x to 2.25x.
The bigger story is FP4. Blackwell's fifth-generation Tensor Cores add a native 4-bit floating-point format that Hopper simply does not have. At FP4 the B200 reaches roughly 9 PFLOPS dense. For inference workloads that tolerate 4-bit quantization (and a growing number do, with careful calibration), this roughly doubles effective throughput again versus running the same model at FP8, and it has no H200 equivalent at all. When you see marketing multipliers larger than 2x, they almost always come from comparing B200 FP4 against Hopper FP8. That is a legitimate real-world path for many inference deployments, but it is not an apples-to-apples precision comparison, so read the footnotes.
For FP64 (double-precision HPC and scientific simulation), the two are close, roughly 37 vs 34 TFLOPS. If your workload is classical HPC rather than AI, neither part is a dramatic upgrade over the other, and that should shape your decision.
Independent MLPerf-based comparisons line up with the spec-sheet math. Reported per-GPU results include roughly 2x on GPT-3 pre-training and roughly 2.2x on Llama 2 70B fine-tuning for B200 over H200, and roughly 2.5x on single-GPU inference measured in tokens per second. At the system level the gap widens: an 8-GPU B200 node has shown roughly 3.1x the throughput of eight H200s on the Llama 2 70B interactive inference benchmark, because faster NVLink and more memory compound across the node. NVIDIA's own rack-scale claims (30x inference, 4x training) are measured against H100, not H200, and against the full GB200 NVL72 system rather than a single GPU, so they are not the right yardstick for an 8-GPU box.
NVLink and node design: HGX B200 vs GB200 NVL72
Single-GPU specs only tell half the story, because these chips are almost always deployed in groups where the interconnect decides how well they scale.
NVLink bandwidth doubles from 900 GB/s per GPU on the H200 to 1.8 TB/s per GPU on the B200. Inside a server, that is the fabric that carries tensor-parallel and expert-parallel traffic between GPUs, so doubling it directly benefits large models split across multiple devices and mixture-of-experts routing.
At the node level there are two dominant Blackwell designs, and they target different buyers:
-
HGX B200 (8-GPU): the direct successor to the familiar 8-way HGX/DGX H200 box. Eight B200s on an NVSwitch baseboard give roughly 1.44 TB of aggregate HBM3e, about 64 TB/s of combined memory bandwidth, roughly 36 PFLOPS FP8 dense and 72 PFLOPS FP4 dense across the node, with a 14.4 TB/s NVLink domain. Total board power lands around 10 kW and up. This slots into existing 8-GPU server designs and is available in both air-cooled and liquid-cooled variants, which makes it the pragmatic upgrade path for most data centers.
-
GB200 NVL72: a rack-scale, liquid-cooled system that fuses 72 Blackwell GPUs and 36 Grace CPUs into a single NVLink domain of 130 TB/s, with 13.4 TB of HBM3e at 576 TB/s aggregate. The whole rack behaves like one very large accelerator. This is the platform behind NVIDIA's headline "30x faster trillion-parameter inference" and "4x faster training" figures versus H100, and its "25x more performance at the same power" claim. It is aimed at frontier-scale training and the largest inference deployments, and it is liquid-cooled by design, not by option.
The H200, by contrast, lives almost entirely in the mature 8-GPU HGX/DGX world, which is precisely why it is easy to buy, rack, and cool today.
Power, cooling, and deployment
This is where the decision often gets made on the facilities side rather than the spec sheet.
The H200 at 700 W per GPU fits comfortably in air-cooled 8-GPU chassis that most colocation and enterprise data centers already run. An 8-way H200 node is a known, well-supported quantity. If your facility is air-cooled and you do not want to touch your cooling plant, this is the frictionless path.
The B200 at roughly 1,000 W per GPU pushes an 8-GPU node past 10 kW just for the accelerators, before CPUs, networking, and storage. Air cooling a B200 node is possible in specific configurations, but the sweet spot, and the required design for GB200 NVL72, is direct-to-chip liquid cooling. Liquid cooling brings higher density and better energy efficiency, but it also brings CDUs, manifolds, leak detection, and facilities work that many sites have not yet deployed. That retrofit cost and lead time is a real part of the B200 total cost, and it is easy to leave out of a naive price comparison.
Size your power and thermal budget before you commit, using the power calculator and the cooling calculator.
Availability, price, and the rent-vs-buy question
The H200 is shipping in volume and is the more available, lower-cost, lower-risk option today. The B200 carries a meaningful premium per GPU and, in many regions, longer lead times, though supply has improved through 2026. We do not quote fixed street prices here because they move week to week and vary by vendor, region, and volume. Get a current quote rather than trusting a stale number.
The more useful framing is total cost of ownership, not sticker price. A B200 that finishes a training run in roughly half the wall-clock time, or serves roughly twice the tokens per GPU, can be cheaper per unit of work even at a higher purchase price, especially once you factor energy efficiency per token. But that only pays off if you keep the hardware highly utilized. A B200 sitting at 30% utilization is an expensive way to underuse a premium chip, and its liquid-cooling overhead makes idle time hurt more.
That points to a simple rule of thumb on rent vs buy:
- Buy when utilization is high and sustained (think steady training pipelines or 24/7 inference at scale). Owned hardware amortizes well above roughly 60 to 70% utilization, and you capture the efficiency gains directly. Weigh purchase price, power, cooling retrofit, rack space, and staff against your expected duty cycle.
- Rent when demand is bursty, when you want to validate B200 performance on your own models before a capital commitment, or when you need capacity now without a facilities project. Renting also lets you A/B the two parts on your actual workload, which is worth more than any benchmark table.
A common and sensible pattern is a mixed fleet: H200 for bulk inference and fine-tuning where its throughput-per-dollar and easy air cooling shine, and B200 (or GB200 NVL72) reserved for the largest training runs and latency-critical, FP4-quantized inference.
Model your own break-even with the rent-vs-buy calculator.
Verdict
Buy the B200 if you are training or serving at or near frontier scale, you can deliver liquid cooling (or you are going straight to GB200 NVL72), and you will keep the GPUs highly utilized. You get roughly 2x to 2.25x the dense throughput, 36% more memory, double the NVLink bandwidth, and native FP4 that can double effective inference throughput again. For the largest workloads, it is the better cost-per-token even at a premium price.
Buy the H200 if you want proven Hopper hardware that ships today, drops into air-cooled 8-GPU nodes, and delivers the best throughput-per-dollar for the money right now. It remains an excellent bulk inference and fine-tuning workhorse, and it avoids a cooling retrofit entirely.
For many teams the answer is both: H200 for the steady, high-volume base load, and B200 for the frontier runs and FP4 inference where the newer architecture earns its premium.
FAQ
Is the B200 really 2x faster than the H200? For dense throughput at matched precision (FP8 or BF16), yes, roughly 2x to 2.25x, and independent MLPerf-based results back that up at around 2x to 2.5x per GPU depending on the workload. Larger multipliers you see quoted usually compare B200 FP4 against Hopper FP8, or compare full rack-scale GB200 NVL72 systems against H100, so check what is actually being measured.
Does the H200 use more memory than the H100? Yes. The H200 keeps the H100's Hopper compute but upgrades to 141 GB of HBM3e at 4.8 TB/s, versus 80 GB at roughly 3.35 TB/s on the H100. That extra capacity and bandwidth is the entire point of the H200, and it is why it generates tokens faster than an H100 despite near-identical FLOPS.
Does the B200 need liquid cooling? Not strictly for the 8-GPU HGX B200, which is available in air-cooled and liquid-cooled variants, though at roughly 1,000 W per GPU liquid cooling is the recommended and denser path. The rack-scale GB200 NVL72 is liquid-cooled by design. The H200 at 700 W runs fine on standard air cooling.
Which is better value for inference? It depends on scale and precision. For steady, high-volume inference on air-cooled infrastructure, the H200 often wins on throughput-per-dollar today. For latency-critical or very large-model inference where you can use FP4 quantization and keep utilization high, the B200's bandwidth and native FP4 can make it cheaper per token despite the higher price. Model it on your own traffic with the rent-vs-buy calculator before committing.