What Are Model Weights?

What Are Model Weights?

When people talk about AI models, they often mention weights. But what are they exactly? In the simplest terms, model weights are the memory of everything the AI has learned during training. They are numbers, billions of them, stored in a file that defines the model's behavior.

Think of weights as the knobs and dials on a giant mixing console. When the model is first created, all the knobs are set to random positions. The model knows nothing. As training progresses, each knob gets adjusted little by little. A knob that helps the model give correct answers gets turned up. A knob that leads to wrong answers gets turned down. After training on billions of examples, the knobs settle into positions that make the model accurate.

Each weight is just a number, typically a floating point value like 0.342 or -1.157. A single weight by itself means nothing. But when you combine millions or billions of them in the right structure, they encode patterns, relationships, and knowledge. The weights of a language model contain grammar rules, factual knowledge, reasoning patterns, and even stylistic preferences, all stored as numerical values.

This is why model weights are so valuable. Training a model from scratch costs millions of dollars in computing power. The weights are the result of that investment. When you download an open source model like Llama or Mistral, you are downloading the weights. The architecture of the model is just the blueprint, the weights are the actual knowledge.

Model weights also determine the size of a model. A 7 billion parameter model has 7 billion weights. Each weight needs to be stored in memory, which is why larger models need more VRAM. When you quantize a model, you are reducing the precision of these weights, making them take less space at the cost of a small amount of accuracy.