GGUF, EXL2, GPTQ, and AWQ: A Comprehensive Comparison for LLM Serving

Learn the key differences between GGUF, EXL2, GPTQ, and AWQ quantization formats and how they are used in llama.cpp and vLLM for home and enterprise users.

Introduction

Modern large language models (LLMs) are extremely memory-hungry. A 70-billion parameter model in full precision (FP16) requires around 140 GB of VRAM—far beyond even the most powerful consumer GPUs. Quantization reduces the precision of weights from 16 bits down to 4 bits or lower, drastically cutting memory requirements. But not all quantization formats are created equal. The choice between GGUF, EXL2, GPTQ, and AWQ depends on your hardware, your serving engine, and your priorities: portability, speed, or multi-user throughput.

Overview of Quantization Formats

GGUF (GPT-Generated Unified Format) is the native format of llama.cpp and the most versatile. It runs on CPUs, Apple Silicon, mixed CPU/GPU setups, and supports a wide range of quantization levels (2 to 8 bits). It is the standard for local, single-user, and edge deployments. GPTQ (Generalized Post-Training Quantization) was the long-standing GPU standard. It is optimized for NVIDIA GPUs and offers a good balance between compression and inference speed, but it is increasingly replaced by newer formats like AWQ. AWQ (Activation-aware Weight Quantization) is the current industry standard for GPU production. It protects salient weights based on activation patterns, which improves accuracy at the same bit-width. It is highly optimized for modern NVIDIA GPU kernels (Marlin) and ideal for high-throughput serving. EXL2 is the format of the ExLlamaV2 engine. It offers the finest control over bit-width (e.g., 3.5, 4.25, 5.0 bits per weight) and allows you to precisely fit a model into limited VRAM. It is often the fastest format for single-user generation on NVIDIA consumer GPUs.

Comparison Table of Formats

Usage in llama.cpp

llama.cpp is built around the GGUF format. It is the primary engine for users who do not have a dedicated high-end NVIDIA GPU, need to run models on CPUs or Apple Silicon, or require a one-size-fits-all solution that is easy to set up. With llama.cpp, you can offload parts of a model to the GPU while keeping the rest in system RAM (CPU+GPU offloading). This allows running large models even on devices with limited VRAM. Tools like Ollama and LM Studio use llama.cpp under the hood, making GGUF the easiest choice for beginners. For local chat applications, text generation on a laptop, or edge devices, GGUF is the undisputed champion.

Usage in vLLM

vLLM is designed for high-throughput production environments. It implements PagedAttention and continuous batching to serve many users simultaneously. Natively, vLLM supports AWQ and GPTQ. These are the formats you should use when deploying vLLM in production. AWQ is preferred due to its superior accuracy and speed (especially with the Marlin kernel). GPTQ is still supported, but for new projects, AWQ is recommended. GGUF is only experimentally supported in vLLM and is not suitable for production—it is often slower and less optimized. EXL2 is not supported by vLLM; it is exclusive to the ExLlamaV2 engine, which is optimized for single-user latency.

Choosing the Right Format

For home users or local development: Use GGUF with llama.cpp (or tools like Ollama). It is the most compatible, runs on almost any hardware, and is easy to manage. For enterprise or high-throughput serving: Use AWQ with vLLM on NVIDIA GPUs. It provides the best balance of speed, accuracy, and scalability for serving multiple concurrent users. For power users (single-user speed): If you have an NVIDIA GPU and want the absolute fastest generation speed for a single user, use EXL2 with the ExLlamaV2 engine. Always check that your serving engine supports the chosen format before quantizing a model.

Conclusion

The choice of quantization format is a strategic decision based on your use case. GGUF dominates the local space, AWQ leads in production, EXL2 rules the high-speed niche, and GPTQ remains a solid but older option. By understanding these differences, you can optimize your LLM serving for both home and enterprise needs.