A GPU name is not a complete SKU
Architecture, package, memory, host link, scale-up fabric, power, cooling, partition mode, and software matrix all belong in the comparison.
- Match scope.Per chip, module, server, or rack; one-way or bidirectional; dense or sparse.
- Match precision.BF16, vendor FP8 variants, INT8, NVFP4, and MXFP4 are not interchangeable labels.
- Match package.PCIe, SXM, OAM, NVL, Superchip, and rack systems can change memory, link, power, and operating requirements.
- Match software.Pin driver, framework, serving engine, kernels, model revision, and quantization format.
- Match workload.Report the input/output distribution, concurrency, SLO, and measurement boundary.
Compare memory, movement, envelope, and fit together
These are representative variants, not every regional or export-control SKU. Follow the linked live specification before procurement.
L4
- Memory
- 24 GB GDDR6
- Bandwidth
- 300 GB/s
- Package
- 72 W · low-profile PCIe
- Fabric
- PCIe Gen4 · no NVLink
- Formats
- FP8 / BF16 / FP16 / INT8
Best starting fit: Low-power inference, video and AV1, smaller models
Check: No MIG; memory is the hard boundary.
L40S
- Memory
- 48 GB GDDR6 ECC
- Bandwidth
- 864 GB/s
- Package
- 350 W · dual-slot PCIe
- Fabric
- PCIe Gen4 · no NVLink
- Formats
- FP8 / BF16 / FP16 / INT8
Best starting fit: Single-GPU generative AI, media, rendering, replicas
Check: Poorer fit for communication-heavy model parallelism.
A100 80 GB SXM
- Memory
- 80 GB HBM2e
- Bandwidth
- 2.039 TB/s
- Package
- 400 W · SXM
- Fabric
- NVLink up to 600 GB/s
- Formats
- TF32 / BF16 / FP16 / INT8
Best starting fit: Mature training, FP64 HPC, partitioned services
Check: No native FP8; PCIe and 40 GB variants differ materially.
H100 SXM
- Memory
- 80 GB HBM3
- Bandwidth
- 3.35 TB/s
- Package
- Up to 700 W · SXM
- Fabric
- NVLink up to 900 GB/s
- Formats
- FP8 / BF16 / TF32 / FP16
Best starting fit: High-compute training and latency-sensitive inference
Check: H100 NVL has 94 GB and a different power/fabric profile.
H200 SXM
- Memory
- 141 GB HBM3e
- Bandwidth
- 4.8 TB/s
- Package
- Up to 700 W · SXM
- Fabric
- NVLink up to 900 GB/s
- Formats
- FP8 / BF16 / TF32 / FP16
Best starting fit: Memory- and KV-bound inference, larger batches and context
Check: A memory-system refresh, not a new compute architecture.
B200 SXM
- Memory
- 180 GB HBM3e
- Bandwidth
- Up to 8 TB/s
- Package
- Up to 1,000 W · SXM6
- Fabric
- NVLink up to 1.8 TB/s
- Formats
- NVFP4 / FP8 / BF16 / FP16
Best starting fit: Large-model training and low-precision inference
Check: Only valuable when the chosen software has optimized Blackwell kernels.
MI300X
- Memory
- 192 GB HBM3
- Bandwidth
- 5.3 TB/s
- Package
- 750 W · OAM
- Fabric
- 8-GPU UBB Infinity Fabric
- Formats
- FP8 / BF16 / FP16 / FP64
Best starting fit: Large-memory training and single-GPU inference
Check: Validate the exact ROCm, model, kernel, and platform matrix.
MI325X
- Memory
- 256 GB HBM3e
- Bandwidth
- 6 TB/s
- Package
- 1,000 W · OAM
- Fabric
- 8-GPU UBB Infinity Fabric
- Formats
- FP8 / BF16 / FP16 / FP64
Best starting fit: Very large models, KV capacity, memory-bound workloads
Check: Memory refresh over MI300X; do not assume a compute-architecture jump.
MI350X
- Memory
- 288 GB HBM3e
- Bandwidth
- 8 TB/s
- Package
- 1,000 W · OAM
- Fabric
- 7 scale-up + 1 scale-out IF links
- Formats
- MXFP4/6/8 / OCP FP8 / BF16
Best starting fit: Large-memory training, inference, and HPC
Check: Partition support depends on the current firmware/software matrix.
MI355X
- Memory
- 288 GB HBM3e
- Bandwidth
- 8 TB/s
- Package
- 1,400 W · OAM
- Fabric
- 7 scale-up + 1 scale-out IF links
- Formats
- MXFP4/6/8 / OCP FP8 / BF16
Best starting fit: Highest-throughput CDNA 4 deployments
Check: Power and liquid/high-density cooling economics are first-order constraints.
Gaudi 3 OAM
- Memory
- 128 GB HBM2e
- Bandwidth
- 3.7 TB/s
- Package
- 900 W · OAM
- Fabric
- 24 × 200 GbE RoCE v2 ports
- Formats
- FP8 / BF16 / FP16 / TF32
Best starting fit: Open-Ethernet scale-out evaluation
Check: Uses HPU/SynapseAI, not CUDA; prove model and operator coverage first.
The suffix often changes the system more than the generation
| Variant | What can change | Choose it when |
|---|---|---|
| PCIe card | Host link, slot count, power ceiling, cooling, sometimes memory | Independent replicas, graphics/media, existing servers, lower power density |
| SXM / OAM module | Higher power, HBM, scale-up links, platform-specific baseboard | Training or model-parallel inference inside a tightly connected node |
| NVL / bridge pair | Paired memory and link behavior, different host and power envelope | Models that benefit from a small, directly linked multi-GPU domain |
| HGX / UBB 8-GPU platform | Topology, aggregate memory, CPU/NIC ratio, chassis and cooling | Scale-up jobs that communicate every layer or optimizer step |
| Superchip / rack system | CPU coupling, rack fabric, facility power and liquid-cooling obligations | Large training, MoE, or inference fleets that can use the whole fabric domain |
- Request the exact manufacturer part and server bill of materials - not “H100 class.”
- Record memory per GPU, GPU count, link topology, CPU/RAM, local storage, NIC speed, power cap, and cooling mode.
- Ask whether quoted link bandwidth is per link, per GPU, aggregate, and bidirectional.
- Confirm that the cloud instance exposes the advertised peer links to the guest.