New Techniques in AI

New Techniques in AI

AI research moves incredibly fast, and new techniques are constantly improving how models are built and trained. Understanding these innovations helps you appreciate how the field is evolving and what to expect from future models.

Grouped Query Attention, or GQA, is an improvement over standard multi-head attention. Instead of each attention head having its own keys and values, groups of heads share them. This reduces the KV cache size significantly without sacrificing quality. GQA is used in Llama 2 and 3, Mistral, and many other modern models. It is a key reason why these models can handle long contexts efficiently.

Multi-Query Attention, or MQA, goes even further by having all attention heads share a single set of keys and values. This reduces memory usage even more but can slightly reduce quality. Models like Falcon and PaLM use MQA. The trend is toward GQA as a good middle ground that balances memory efficiency and model quality.

Rotary Position Embedding, or RoPE, has become the standard way to encode position information in transformers. Unlike older methods that added position information to the input, RoPE modifies the attention mechanism itself to account for position. This allows models to handle variable length sequences more naturally and has enabled the extension of context windows to hundreds of thousands of tokens.

Flash Attention is a software technique that reorders the attention computation to be much more memory efficient. Instead of storing the full attention matrix, which scales quadratically with context length, Flash Attention computes attention in blocks that fit in fast on chip memory. This can reduce memory usage by 10x or more, making long context windows practical. It is now standard in all major training and inference frameworks.