Prompt Processing and GPU Speed

Prompt Processing and GPU Speed

When you send a prompt to an AI model, two distinct phases happen. First, the model processes your input prompt, which is called prompt processing or prefill. Then it generates the response tokens one at a time. These two phases have very different performance characteristics and are limited by different factors.

During prompt processing, the model takes your entire input and runs it through the network in parallel. If your prompt is 1000 tokens long, those 1000 tokens are processed simultaneously. This phase is compute bound, meaning it is limited by how fast the GPU can do matrix multiplications, not by memory bandwidth. A GPU with more Tensor Cores and higher clock speeds will process prompts faster.

This is why prompt processing speed scales with GPU compute power. The H100 with its massive number of Tensor Cores can process prompts much faster than a consumer GPU, even if the consumer GPU has similar memory bandwidth. For short prompts, this phase is nearly instant. For very long prompts, like analyzing a 100 page document, prompt processing can take several seconds even on fast hardware.

After the prompt is processed, the model switches to token generation mode. This is memory bandwidth bound, as we discussed earlier. Each token requires loading all the model weights again. The GPU's compute units spend most of their time waiting for data to arrive from memory. This is why token generation speed depends on memory bandwidth, not compute power.

This split explains some counterintuitive behavior. A GPU might process prompts very quickly but generate tokens slowly if it has lots of compute but limited bandwidth. Conversely, a GPU with modest compute but high bandwidth, like some older server GPUs, might generate tokens quickly but take a long time to process long prompts. Understanding which phase matters more for your use case helps you choose the right hardware.