Token Generation Speed and Bandwidth

Token Generation Speed and Bandwidth

Token generation is the phase where the model produces output, one token at a time. This is what you see when a chatbot types out its response character by character. Understanding what determines this speed is crucial for choosing hardware and setting expectations.

The fundamental equation is simple: token speed equals memory bandwidth divided by model size. If your GPU has 1 TB per second of bandwidth and your quantized model is 7 GB in size, your maximum theoretical speed is about 140 tokens per second. In practice, you get about 70 to 80 percent of this due to overhead, but the equation holds.

Let us look at some real examples. An RTX 4090 has about 1 TB per second of bandwidth. Running a 7B model in 4-bit, which is about 4 GB, gives a theoretical maximum of 250 tokens per second. Running a 70B model in 4-bit, which is about 40 GB, gives only 25 tokens per second. The same hardware, but a larger model means slower generation.

This is why batch size matters for inference servers. If you process multiple prompts at the same time, the weights only need to be loaded once and can be shared across all the prompts in the batch. This dramatically increases total throughput, measured in tokens per second across all users, at the cost of higher latency for each individual request.

For interactive use like chatbots, you want low latency, which means small batches or single user inference. For batch processing like analyzing thousands of documents, you want high throughput, which means large batches. Understanding this distinction helps you decide whether to run models locally or use an API. Local is great for latency sensitive interactive use, while APIs excel at high throughput batch processing.