Mixtral 8x7B outperforms or matches Llama 2 70B and GPT-3.5 across various benchmarks.

In the ever-evolving landscape of natural language processing, the pursuit of more powerful and versatile language models has led to remarkable breakthroughs. Among these, Mixtral 8x7B stands tall as a Sparse Mixture of Experts (SMoE) language model, showcasing a paradigm shift in performance and efficiency. This cutting-edge model, built upon the foundation of Mistral 7B, introduces a novel architecture with eight feedforward blocks (experts) per layer, revolutionizing the way tokens are processed.

With a keen focus on optimizing parameter usage, Mixtral 8x7B provides each token access to an impressive 47 billion parameters, all while utilizing a mere 13 billion active parameters during inference. Its unique approach, where a router network dynamically selects two experts for each token at every layer, allows for unparalleled adaptability and responsiveness.

Under the Hood: Mixtral 8x7B Architecture

Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model that has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference.

Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. The model architecture parameters are summarized in Table 1, and a comparison of Mixtral with Llama is provided in Table 2. Mixtral outperforms or matches Llama 2 70B performance on almost all popular benchmarks while using 5x fewer active parameters during inference.

Here’s why Mixtral is special:

• It’s very good at different tasks like math, coding, and languages.
• It uses less power than other similar models because it doesn’t have to use all its experts all the time.
• This makes it faster and more efficient.

Think of it like this:

• You need to solve a math problem and a coding problem.
• Mixtral picks the math expert for the math problem and the coding expert for the coding problem.
• They both work on their tasks and give you the answers, but you only talk to them one at a time.
• Even though you don’t see all 8 experts all the time, they’re all ready to help if needed.

Benchmark Performances

The benchmark performances of the Mixtral 8x7B model, a Sparse Mixture of Experts (SMoE) language model, are compared to Llama 2 70B and GPT-3.5 across various tasks. Mixtral outperforms or matches Llama 2 70B and GPT-3.5 on most benchmarks, particularly in mathematics, code generation, and multilingual understanding.

It uses a subset of its parameters for every token, allowing for faster inference speed at low batch-sizes and higher throughput at large batch-sizes. Mixtral’s performance is reported on tasks such as commonsense reasoning, world knowledge, reading comprehension, math, and code generation. It is observed that Mixtral largely outperforms Llama 2 70B on all benchmarks, except on reading comprehension benchmarks, while using 5x fewer active parameters. Detailed results for Mixtral, Mistral 7B, Llama 2 7B/13B/70B, and Llama 1 34B2 are provided, showing that Mixtral outperforms or matches Llama 2 70B performance on almost all popular benchmarks while using 5x fewer active parameters during inference.

Impressive Retrieval

The retrieval accuracy of the Mixtral model is reported to be 100% regardless of the context length or the position of the information in the sequence. The model is able to successfully retrieve information from its context window of 32k tokens, regardless of the sequence length and the location of the information in the sequence.

Licensing and Open Source Community

Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model that is licensed under the Apache 2.0 license, making it free for academic and commercial usage, ensuring broad accessibility and potential for diverse applications. The model is released with open weights, allowing the community to run Mixtral with a fully open-source stack. The French startup recently raised $415M in venture funding and has one of the fastest-growing open-source communities.

It’s worth noting that the details regarding the data used for pre-training and the specific loss function employed are conspicuously absent from the available information. This omission leaves a gap in our understanding of the model’s training process. There is no mention of whether any additional loss for load balancing is being utilized, which could provide valuable insights into the model’s optimization strategy and robustness. Despite this gap, the outlined architectural and performance characteristics of Mixtral 8x7B offer a compelling glimpse into its capabilities and potential impact on the field of natural language processing.