Dense vs MoE Models

Dense vs MoE Models

Not all AI models are built the same way. Two of the most important architectures you will hear about are dense models and mixture of experts, or MoE, models. They take very different approaches to solving the same problem, and each has its own strengths and weaknesses.

A dense model activates all of its parameters for every input. If a model has 7 billion parameters, every single one of them contributes to processing every token. This is simple and straightforward: the whole brain works on every problem. Dense models like Llama and Mistral are well understood and tend to be more predictable in their behavior.

A mixture of experts model takes a modular approach. Instead of one giant network, it has many smaller networks called experts, plus a router that decides which experts to use for each input. When you give the model a prompt, the router looks at it and sends it to only a few of the experts, maybe two or four out of a hundred. The other experts sit idle.

The advantage of MoE is efficiency. A dense model with 100 billion parameters uses all 100 billion for every calculation. An MoE model might have 100 billion parameters total, but only activates 10 billion for each token. This means it can be much faster per token than a dense model of the same total size, while still having the knowledge capacity of the larger model.

There are trade offs though. MoE models require more memory because all experts need to be loaded even if only a few are used. They can also be harder to train and more unpredictable, since different experts might give different answers to similar questions. Models like Mixtral 8x7B and GPT-4 use MoE architectures, while Llama and Falcon use dense architectures.