Qwen 3.8 27B: 4-bit Quantization, vLLM Tensor Parallelism, and VRAM/Context Analysis

Qwen 3.8 27B: Deployment Guide with vLLM and Tensor Parallelism

Qwen 3.8 27B is a dense transformer model with 27 billion parameters, 64 layers, and a hidden dimension of 5120. It uses Grouped-Query Attention (GQA) to reduce KV cache memory, making it suitable for long-context inference on multi-GPU setups. In this article, we explain how to load a 4-bit quantized version of the model using vLLM with tensor parallelism, calculate VRAM requirements for weights and KV cache, and estimate the maximum context length achievable on two RTX 3090 and two RTX 5090 GPUs.

Loading Qwen 3.8 27B on vLLM with Tensor Parallelism

vLLM supports tensor parallelism (TP) to distribute the model across multiple GPUs. With TP size 2, each GPU holds half of the model weights and computes half of the operations per layer, synchronizing via NCCL. The command below starts an OpenAI-compatible server with a 4-bit quantized model (e.g., AWQ or GPTQ) and TP=2:

python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3.8-27B --tensor-parallel-size 2 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8

The --gpu-memory-utilization flag controls the fraction of VRAM used for weights, KV cache, and CUDA context. Always choose the smallest TP size that fits the model to minimize communication overhead. If the model fits on a single GPU, avoid TP and run separate instances for higher throughput.

VRAM Usage: Model Weights and KV Cache

VRAM is consumed by two main components: model weights and KV cache. For a 4-bit quantized model, each parameter occupies approximately 0.5 bytes (plus a small overhead for scales and zeros). The weight footprint is calculated as follows:

The KV cache size depends on the context length, precision, and model architecture. For Qwen 3.8 27B with GQA, the formula is: 2 * num_layers * num_kv_heads * head_dim * seq_len * bytes_per_element. With 64 layers, 8 KV heads, and head_dim 128, we can compute the KV cache for a 32k context.

Max Context Calculation on 2x RTX 3090 and 2x RTX 5090

Assuming a total VRAM of 48 GB for two RTX 3090 (24 GB each) and 64 GB for two RTX 5090 (32 GB each), we subtract the weight footprint (~15 GB) and CUDA overhead to find the VRAM available for KV cache. The max context is then estimated by dividing the available VRAM by the KV cache size per token.

For BF16 KV cache, the max context would be roughly half of the FP8 values, i.e., ~150k tokens on 2x 3090 and ~230k tokens on 2x 5090. These numbers assume a batch size of 1 and no other memory overhead. In practice, vLLM reserves some memory for CUDA graphs and scheduling, so the actual context may be slightly lower.

FP8 vs BF16 for KV Cache and FP8 Storage Types

BF16 offers higher precision but consumes twice the memory of FP8. For inference, FP8 is the preferred choice because it doubles the KV cache capacity with negligible accuracy loss. Within FP8, there are two storage formats: E4M3 and E5M2. E4M3 provides a better balance of range and precision and is recommended for KV cache and weights. E5M2 has less precision and is typically used for gradients in training. In vLLM, you can enable FP8 KV cache with --kv-cache-dtype fp8.

In conclusion, Qwen 3.8 27B with 4-bit quantization can be efficiently deployed on dual-GPU systems. Using FP8 KV cache maximizes context length and throughput. On 2x RTX 3090, you can handle ~300k tokens of context, while 2x RTX 5090 offers ~460k tokens, making both setups excellent for long-document processing and high-concurrency serving.