Mixtral of Experts

Abstract

We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.

arXiv commentSee more details at https://mistral.ai/news/mixtral-of-experts/

Cited by in corpus (11)

• The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges
• OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models
• Learning from models beyond fine-tuning
• DirectMultiStep: Direct Route Generation for Multistep Retrosynthesis
• Transforming Agency. On the mode of existence of Large Language Models
• Large Language Models for Combinatorial Optimization: A Systematic Review
• Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis
• Evidence Is All You Need: Ordering Imaging Studies via Language Model Alignment with the ACR Appropriateness Criteria
• Cache Management for Mixture-of-Experts LLMs -- extended version
• CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory
• A Roadmap for Tamed Interactions with Large Language Models