Memory Bandwidth: The AI Speed Limit

Memory Bandwidth: The AI Speed Limit

When people talk about AI speed, they often focus on the processor. But for most AI workloads, especially inference, the processor is not the bottleneck. The real speed limit is memory bandwidth. Understanding this is key to understanding why some hardware runs AI faster than others.

Memory bandwidth is how fast data can be moved from memory to the processor. Think of it like a water pipe. A wider pipe lets more water flow at once. In AI terms, every token the model processes requires moving all the model weights from memory to the compute units. For a 7 billion parameter model, that means moving 14 GB of data for every token.

This is why memory bandwidth matters so much. The GPU can do matrix multiplications incredibly fast, but it has to wait for the weights to arrive before it can start computing. If the memory can only deliver data at 500 GB per second, and you need to move 14 GB for each token, you cannot generate tokens faster than about 35 tokens per second, regardless of how fast your GPU cores are.

Different hardware has vastly different memory bandwidth. A consumer GPU like an RTX 4090 has about 1 TB per second of bandwidth. A high end data center GPU like the H100 has about 3.35 TB per second. But even the fastest GPUs are limited by bandwidth. This is why improvements in memory technology, like HBM3 and GDDR7, directly translate to faster AI performance.

The relationship is simple: for LLM inference, speed is proportional to bandwidth divided by model size. Double the bandwidth, double the speed. Halve the model size through quantization, double the speed. This is why quantization is so powerful: it reduces the amount of data that needs to be moved, effectively increasing the speed of your existing hardware.