Floating Point Formats: BF16, FP8, INT8

Floating Point Formats: BF16, FP8, INT8

AI models store their weights and perform calculations using different number formats. The format you choose affects how much memory the model uses, how fast it runs, and how accurate its outputs are. The most common formats today are FP16, BF16, FP8, and INT8.

FP16, or 16-bit floating point, has been the standard for AI training and inference for years. Each number uses 16 bits, with a certain number of bits for the exponent and mantissa. It offers good precision and is widely supported by GPUs. A 7B model in FP16 needs about 14 GB of memory.

BF16, or Brain Floating Point 16, was developed by Google and is now supported by most modern GPUs. It uses the same 16 bits as FP16 but allocates more bits to the exponent and fewer to the mantissa. This means it can represent a wider range of numbers, which is useful for training, but with slightly less precision. In practice, BF16 and FP16 give very similar results, and BF16 is often preferred for training because it is more stable.

FP8 is a newer format that uses only 8 bits per number. It comes in two variants: E4M3, which prioritizes range, and E5M2, which prioritizes precision. FP8 is primarily used during inference on the latest generation of GPUs like the H100 and Blackwell. It cuts memory and bandwidth requirements in half compared to FP16 while maintaining surprisingly good quality for inference.

INT8, or 8-bit integer, is a format used in quantization. Instead of floating point numbers with exponents and mantissas, each value is a simple integer. INT8 requires careful calibration to map the original floating point values to integers without losing too much information. It offers the same memory savings as FP8 but generally requires more careful tuning to maintain quality.