専門家の混合

Mixture of Experts — Grokipedia Fact-checked by Grok 2 months ago Mixture of Experts Ara Eve Leo Sal 1x Mixture of Experts (MoE) is a machine learning architecture that employs multiple specialized sub-models, known as "experts," and a gating or routing mechanism to selectively activate only the relevant experts for each input. This enables efficient conditional computation, as only a subset of the model's total parameters is activated per token or input, allowing for substantial scaling in model capacity with reduced computational overhead compared to dense architectures. In modern large language models (LLMs), sparse MoE designs dominate frontier models due to their energy efficiency advantages, enabling parameter counts in the hundreds of billions to trillions while maintaining lower per-token energy use, superior performance per watt, and reduced inference costs. [1] [2] [3] In a typical sparse MoE layer within LLMs, a router network dynamically selects a small number of experts (often two) from a larger set (such as eight) to process each token at every layer, with the outputs combined additively. This sparse activation means that inference operates at the speed and cost of a much smaller dense model while accessing the knowledge and capacity of a vastly larger parameter space. For instance, Mixtral 8x7B , a sparse MoE model, has a total of 46.7 billion parameters but activates only 12.9 billion per token, matching or outperforming Llama 2 70B and GPT-3.5 across most benchmarks, with particularly strong results in mathematics, code generation, and multilingual tasks. Its instruction-tuned variant, Mixtral 8x7B Instruct, achieves top performance among open-source models and surpasses GPT-3.5 Turbo , Claude-2.1 , Gemini Pro , and Llama 2 70B chat on human preference benchmarks. [4] [5] More recent frontier models continue this trend. Mistral Large 3 , released as part of the Mistral 3 family, is a sparse MoE model with 675 billion total parameters and 41 billion active parameters, optimized for high-throughput, long-context workloads through hardware-software co-design (including NVIDIA Blackwell attention and MoE kernels). It achieves parity with leading instruction-tuned open-weight models on general prompts, ranks highly on leaderboards such as LMSYS Arena, and excels in multilingual and multimodal tasks, while supporting efficient inference on systems ranging from a single 8×H100 node to large-scale clusters. [3] The primary advantages of MoE in contemporary LLMs stem from its ability to scale capacity dramatically without proportional increases in compute or energy costs during inference. By activating only a fraction of parameters per token, MoE models deliver improved quality versus inference budget trade-offs, making them highly suitable for large-scale deployment. However, challenges include managing routing load balance, ensuring expert specialization and diversity, and addressing potential overheads in very large-scale deployments (such as those requiring offloading techniques). Ongoing research focuses on refining routing algorithms, training stability, and system-level optimizations to maximize these benefits. [1] [6] [2] Overall, sparse Mixture of Experts architectures represent a key innovation in scaling LLMs efficiently, powering many of the most capable open-weight and frontier models while prioritizing performance-per-watt and cost-effectiveness. [3] [4] Introduction Definition and Overview Mixture of Experts (MoE) is a machine learning architecture that combines multiple specialized sub-networks, known as experts, with a gating or routing mechanism to selectively activate relevant experts for each input. [7] [8] The gating network assigns weights to the experts' outputs based on the input, producing a weighted combination that forms the final result. [7] [9] This approach divides the input space into more homogeneous regions, allowing each expert to specialize in distinct patterns or data subsets while the gating mechanism routes inputs to the most appropriate experts. [7] [9] By activating only a subset of the model for any given input, MoE implements conditional computation, enabling greater overall model capacity without a proportional increase in per-example computation. [8] [7] The primary purpose of MoE is to achieve specialization and efficiency: experts learn to handle specific aspects of complex problems, and the selective activation reduces unnecessary computation compared to dense models where all parameters process every input. [8] [9] This design supports scalability in large-scale neural networks. [7] Historical Development The Mixture of Experts (MoE) architecture originated in the early 1990s as a supervised learning framework that combined multiple specialized neural networks to handle complex tasks more effectively than a single monolithic model. The foundational contribution came in 1991 with "Adaptive Mixtures of Local Experts" by Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton, which introduced a system of separate networks (experts) trained to specialize on different subsets of the data, with a gating network that adaptively weighted their outputs to produce the final prediction. [10] In 1992, John B. Hampshire II and Alex Waibel proposed the Meta-Pi network, a multinetwork classifier that integrated specialized time-delay neural network modules into a superstructure to form distributed representations for robust multisource pattern recognition, such as multispeaker phoneme classification, demonstrating early ideas of combining expert-like components for improved generalization across varying input sources. [11] A significant extension appeared in 1994 with "Hierarchical Mixtures of Experts and the EM Algorithm" by Michael I. Jordan and Robert A. Jacobs, which organized experts in a tree-structured architecture to apply divide-and-conquer principles and introduced the expectation-maximization (EM) algorithm for efficient parameter estimation in these hierarchical models. These early variants emphasized soft, dense combinations of experts and explored training techniques such as EM to address issues like load balancing and convergence. By the early 2010s, MoE concepts began transitioning into deep learning contexts, with gating mechanisms applied per layer in deep networks without sparsity, as demonstrated in the 2013 work on Deep Mixture of Experts that stacked multiple gating and expert sets for factored representations. [12] The subsequent shift to sparse activation paradigms emerged later in the context of large-scale models. Rise in Large Language Models The resurgence of Mixture of Experts (MoE) in large language models traces back to the introduction of sparse MoE layers in transformer architectures with Google's GShard framework in 2020, which scaled multilingual neural machine translation models to over 600 billion parameters using sparsely-gated MoE and conditional computation. [13] In the following years, MoE gained traction in large language models through pioneering works such as Mixtral 8x7B, which demonstrated the viability of sparse MoE for efficient high-performance inference. Since early 2025, MoE architectures have achieved widespread adoption, with over 60% of frontier model releases incorporating MoE designs and nearly all leading frontier models using MoE. As of early 2026, MoE dominates frontier large language models, particularly for its energy efficiency advantages over dense alternatives. [14] [15] The core appeal of MoE in large language models stems from its conditional computation: a routing mechanism activates only a small subset of specialized experts per token, reducing inference energy consumption proportionally to the number of active experts. This sparsity enables models to scale to hundreds of billions of total parameters while engaging only tens of billions actively per token, yielding lower per-token energy use, improved performance per watt, and reduced inference costs compared to dense models. For instance, MoE models achieve significantly greater intelligence per unit of energy and capital invested, with hardware optimizations delivering up to 10x leaps in performance per watt and 10x reductions in cost per token on systems like the NVIDIA GB200 NVL72. [14] Representative examples include Mistral Large 3, released in December 2025, which employs a sparse MoE architecture with 675 billion total parameters but only 41 billion active during inference, achieving top-tier accuracy and energy efficiency across multilingual and multimodal tasks. [16] Core Architecture Experts and Specialization In Mixture of Experts (MoE) architectures, the experts are the specialized sub-networks that perform the bulk of computation in sparse layers. They are typically implemented as feed-forward neural networks (FFNs), often using activation functions such as SwiGLU, as seen in models like Mixtral 8x7B where each expert is a standard feed-forward block matching the architecture of dense transformers. [17] Experts may also take more complex forms in some designs, though FFNs remain the most common. [7] Specialization occurs as experts learn to process distinct subspaces of the input data during training, with the routing mechanism directing tokens to the most suitable experts. This enables different experts to focus on particular patterns or features, such as syntactic elements, rather than broad semantic domains. For instance, in Mixtral 8x7B, analysis of routing patterns across diverse datasets shows that expert selection aligns more strongly with syntax than with high-level topics like mathematics, biology, or philosophy, with consistent routing for tokens like indentation in code or specific keywords such as "self" in Python. [17] Evidence of specialization in practice varies by model and scale. In Mixtral 8x7B, domain-specific patterns are largely absent, with similar expert distributions across topics except for