Understanding Q, K, V Vectors in Transformer Models

In modern artificial intelligence, especially in natural language processing, the Transformer architecture has become the backbone of models like GPT, BERT, and many others. At the core of this architecture lies the attention mechanism, which relies on three key vectors: Query (Q), Key (K), and Value (V). This article will explain how these vectors are derived from tokens, how they are used during prompt processing, and how multi-head attention and all-reduce operations work to produce the final output.

From Words to Tokens and Embeddings

The first step in processing text is to convert each word (or subword) into a token. The vocabulary of the model contains a fixed set of tokens, each with a unique integer ID. Once the input sentence is split into tokens, each token is mapped to a dense vector called an embedding. This embedding is a high-dimensional representation (e.g., 512 or 768 dimensions) that captures the semantic meaning of the token. These embeddings are learned during training and are stored in an embedding matrix.

Deriving Query, Key, and Value Vectors

To enable the attention mechanism, the embedding of each token is transformed into three different vectors: Query, Key, and Value. This is done by multiplying the embedding vector by three learned weight matrices: Wq, Wk, and Wv. These matrices are parameters of the model, updated during training. The resulting vectors have a smaller dimension (often called d_k) compared to the original embedding dimension, which helps in efficient computation.

The Query vector (Q) represents what the token is 'looking for' in the context. The Key vector (K) represents what the token 'offers' as a label. The Value vector (V) contains the actual information that will be aggregated. The attention mechanism determines how much each token should attend to others by comparing queries with keys.

The Self-Attention Mechanism

Once we have Q, K, and V for every token in the sequence, the self-attention mechanism computes the attention scores. For a given token, its query is multiplied (dot product) with the keys of all tokens in the sequence. The resulting scores indicate the relevance of each token to the current one. These scores are then scaled by dividing by the square root of the dimension of the keys (d_k) to stabilize gradients. Next, a softmax function is applied to convert the scores into probabilities that sum to 1. Finally, the probabilities are used to compute a weighted sum of the value vectors, producing a new context-aware vector for the token.

This process is performed in parallel for all tokens in the sequence, making it highly efficient. The new vectors are then passed through a feed-forward neural network and additional layers, gradually building a rich representation of the input.

Prompt Processing and Layer-by-Layer Computation

During prompt processing (the initial encoding of the input), the entire sequence of tokens is processed simultaneously. The embeddings are transformed into Q, K, V, and the self-attention is computed for every token. This is repeated across all layers (typically 12, 24, or more) of the Transformer. Each layer refines the representations, allowing the model to capture complex dependencies such as long-range relationships, syntax, and semantics. The output of the last layer provides a contextualized embedding for each token.

Inference: Adding a New Token

When generating text (inference), the model produces one token at a time. For the new token, only its embedding is computed. Then, its Query is derived. To compute attention, the model needs the Keys and Values of all previous tokens. These are stored in a structure called the KV cache, which avoids recomputing them. The new token's Query is multiplied by all cached Keys, the softmax is applied, and the weighted sum of the cached Values is computed. This produces a new context vector for the new token, which is then passed through the remaining layers and finally a classification layer (softmax over vocabulary) to select the next word.

This incremental approach is efficient because the heavy computations for previous tokens are reused. The KV cache is updated with the new token's Key and Value after each generation step.

Multi-Head Attention and All-Reduce

To capture different types of relationships (e.g., syntactic, semantic, referential) simultaneously, Transformers use multi-head attention. Instead of performing a single attention operation, the model splits the Q, K, and V vectors into multiple smaller heads (commonly 8, 12, or 16). Each head learns to focus on different aspects of the input. The heads work in parallel, each computing its own attention mechanism on a subspace of the vectors.

After each head produces its output, the outputs are concatenated back together into a single vector of the original dimension. This concatenated vector is then multiplied by an output projection matrix (often called Wo) to combine the information from all heads. This final step is equivalent to an all-reduce operation: it merges the parallel computations into a unified representation. The all-reduce ensures that the model benefits from the distributed learning across heads while maintaining a single coherent output for the next layer.

Conclusion

The Query, Key, and Value vectors are fundamental to the attention mechanism in Transformers. They allow the model to dynamically weigh the importance of each token in the context, enabling powerful language understanding and generation. The process of deriving these vectors, computing attention, and using multi-head parallelization with all-reduce is what makes modern AI models so effective. Understanding these concepts is essential for anyone working with or studying large language models.