GPU Memory Bandwidth: HBM and GDDR

GPU Memory Bandwidth: HBM and GDDR

Not all GPU memory is created equal. Consumer graphics cards use GDDR memory, while data center AI accelerators use HBM. The difference in architecture and performance is substantial, and it is one of the main reasons enterprise AI hardware costs so much more.

GDDR, or Graphics Double Data Rate memory, is what you find in consumer GPUs like the RTX 4090. It is designed to be a good balance of speed, capacity, and cost. GDDR6X, used in high end NVIDIA cards, offers around 1 TB per second of bandwidth. The chips are arranged around the GPU on the circuit board, connected by wide memory buses, 384-bit in the case of the RTX 4090.

HBM, or High Bandwidth Memory, is a completely different design. Instead of chips arranged around the GPU, HBM stacks memory chips vertically and places them right next to the GPU die on a silicon interposer. This allows for an incredibly wide memory bus, up to 1024-bit or more. HBM3, used in the H100 and MI300X, offers over 3 TB per second of bandwidth.

The trade off is cost and capacity. HBM is much more expensive to manufacture than GDDR because of the complex stacking and packaging required. HBM also typically offers less total capacity. The H100 has 80 GB of HBM3, while a consumer card could have 24 GB of GDDR6X for a fraction of the price. For data center workloads where every millisecond counts, the extra bandwidth is worth the cost.

The next generation, HBM4 and GDDR7, promise even higher bandwidths. HBM4 targets over 6 TB per second, while GDDR7 will push beyond 1.5 TB per second. These improvements directly translate to faster AI inference, since as we discussed, token generation speed is limited by how fast you can move weights from memory to compute.