What is Model Quantization?

What is Model Quantization?

Model quantization is one of the most important techniques for making AI models practical. It is the process of reducing the precision of the numbers that make up a model's weights, which makes the model smaller, faster, and able to run on less expensive hardware. The trade off is a small reduction in accuracy.

Normally, model weights are stored as floating point numbers. The standard format is 16-bit floating point, where each weight uses 16 bits, or 2 bytes, of memory. A 7 billion parameter model in 16-bit needs 14 GB of memory. If you quantize to 4-bit, each weight uses only 4 bits, or half a byte. That same model now needs only 3.5 GB, which fits easily on a consumer GPU or even in system RAM.

How much accuracy do you lose? Surprisingly little for most tasks. A well quantized 4-bit model can retain 95 to 99 percent of the original model's capability. The quantization process finds the best way to represent the original high precision weights using fewer bits, and modern quantization methods have become very good at minimizing information loss.

There are two main approaches: post training quantization and quantization aware training. Post training quantization, as the name suggests, happens after the model is fully trained. You take the trained weights and compress them. Quantization aware training incorporates the compression into the training process itself, which usually produces better results but requires training the model from scratch.

Quantization is the reason you can run modern AI models on consumer hardware. Without it, running a 70 billion parameter model would require 140 GB of memory, which means multiple expensive data center GPUs. With 4-bit quantization, the same model fits in 35 GB, which a single high end consumer GPU like an RTX 4090 can handle. This has democratized access to powerful AI.