Home › Guides › Mixture of Experts
Tech Explained · 2026Mixture of Experts in 2026: How MoE Routing Works, MoE vs Dense Models and Why 671B Runs Like 37B
Mixture of experts is an LLM architecture that splits each feed-forward layer into many small expert networks and routes every token to only a few of them. DeepSeek-V3 holds 671B parameters but activates just 37B per token, so it answers at the speed of a far smaller model while needing the memory of the full one.
- Active parameters set your speed, total parameters set your bill. Every expert sits in VRAM even when the router ignores it.
- The sparsity is extreme. DeepSeek-V3 computes with about 5.5% of its weights per token, gpt-oss-120b with roughly 4.4%, Mixtral 8x7B with 28%.
- Routing is learned, not assigned. A single linear layer scores every expert per token and the top few win. No one labels "the SQL expert".
- MoE wins on throughput, not on latency at batch size 1. Serve one request at a time and a dense 32B would have been cheaper.
- Self-hosting a frontier MoE is the wrong first move. vLLM's recipe for DeepSeek-V3 is 8x H200. A 120B-class MoE on one 80GB card is the honest start.
- You will meet MoE at the API layer first. Knowing why a model's token price fell while its long-context price did not is the practical payoff.
You read that a new open-weight model has 671 billion parameters, price out a GPU box that could hold it, and quietly close the tab. Then you notice its API price undercuts a 70B model you already use. That gap between the headline parameter count and what a token actually costs is the whole story of mixture of experts, and it is the one architecture shift of the last three years that changed what an AI engineer can realistically run.
What Is Mixture of Experts, in One Pass of a Token
In a standard transformer every token goes through every weight. A 70B dense model does 70 billion parameters' worth of arithmetic for the word "the" and the same for a line of Terraform. Mixture of experts breaks that in one place: the feed-forward network inside each layer.
Instead of one wide feed-forward block, an MoE layer holds many narrow ones called experts plus a tiny router. The router reads the token, scores every expert, and sends it to only the top few. Attention layers are untouched and still run for everything. The model is therefore huge in storage and small in compute per token.
Active parameters vs total parameters
Memorise this distinction, because vendors quote whichever number flatters them. Total parameters are what you store and load. Active parameters are what actually multiply per token, so they drive latency and FLOPs. The Hugging Face model card for DeepSeek-V3 lists 671B total with 37B activated per token at 128K context, and the arXiv technical report (2412.19437) gives the same split plus the layer layout: one shared expert and 256 routed experts, with 8 routed experts active per token.
One token through one MoE layer
The router is the only new component. Everything expensive is the experts it chooses not to run.
Layer layout follows the DeepSeek-V3 technical report (arXiv 2412.19437): one shared and 256 routed experts, 8 routed experts active per token. Checked 6 November 2026.
How MoE Routing Works: Top-k, Shared Experts and Load Balancing
The router is smaller than you expect. Usually a single linear layer maps the token's hidden state to one score per expert, then a softmax over the winners weights their outputs. Two experts scoring 0.9 and 0.1 do not contribute equally.
Top-k: how many experts a token gets
Mixtral 8x7B, which put MoE in front of working engineers, picks 2 of 8. Mistral's figures give it 46.7B total and about 12.9B active at 32,000 tokens of context. The name misleads: 8 times 7 is 56, but shared attention and embeddings mean the real total is 46.7B. Newer models go far sparser. Nebius's model documentation for gpt-oss-120b lists 128 experts with 4 active per token, about 117B total and roughly 5.1B active, Apache 2.0 licensed and quantised to MXFP4 so it fits a single 80GB H100. Qwen3-235B-A22B puts the split in its name: 235B total, 22B active, 128 experts with 8 chosen.
Shared experts and why routers collapse
DeepSeek adds one expert every token always passes through, absorbing the general knowledge that would otherwise be duplicated across 256 specialists. Training also needs a load-balancing term, because a router left alone finds three good experts and sends everything to them, wasting the other 253. That is router collapse, and it is why you cannot bolt MoE onto a dense checkpoint over a weekend.
One correction worth making early, since the vocabulary collides. This routing happens inside one model, per token, in every layer. It is not choosing between whole models per request, which our guide to LLM routing and model routers covers and which interviewers like to see you separate. Transformer internals at this level, attention through expert routing, are what the AI Engineer programme works through live rather than leaving to a reading list.
DeepSeek-V3 at a glance
The four numbers that decide whether you can serve it at all.
Expert counts from arXiv 2412.19437. GPU configuration from vLLM's published DeepSeek-V3 recipe, which lists 8x H200 for the FP8 checkpoint with tensor parallel 8 and expert parallelism. The 671GB figure is a byte-per-parameter estimate at FP8, not a vendor number. Checked 6 November 2026.
MoE vs Dense Models: 10 Criteria That Decide Which One You Serve
Dense is not the losing side. It wins outright in a set of situations that happen to describe most first deployments.
| Criterion | Mixture of experts | Dense model |
|---|---|---|
| Active parameters per token | 4% to 28% of total | 100%, always |
| VRAM you must provision | All experts, routed to or not | All weights, a smaller number |
| Quality per GPU-hour at high batch | Strong, the reason MoE took over open weights | Weaker at the same hardware budget |
| Cost per token at batch size 1 | Poor, huge weights for small compute | Good, nothing idles |
| Single-GPU viability | 120B class at 4-bit only, such as gpt-oss-120b on one 80GB card | Routine to roughly 30B at 4-bit |
| Latency consistency | Variable, expert placement changes the hop count | Predictable |
| Fine-tuning difficulty | Higher, a narrow dataset destabilises the router | Well-trodden, LoRA works as documented |
| Quantisation behaviour | Sensitive, router and shared layers often kept higher precision | Forgiving down to 4-bit |
| Serving stack maturity | Needs expert parallelism, so recent vLLM or SGLang builds | Runs on almost anything |
| Best fit | High-volume batch work, frontier quality on rented hardware | Low-volume, latency-sensitive or on-premise work |
Share of weights that actually compute per token
Lower bar means more speed per parameter stored, and more VRAM spent holding idle experts.
Active divided by total, calculated from the model cards and vendor documentation cited in this guide. Checked 6 November 2026.
Which Open-Weight MoE Models Matter in 2026
Read this as a ladder of hardware requirements, not a quality ranking.
| Model | Total | Active | Experts (total / active) | Context | Practical note |
|---|---|---|---|---|---|
| Mixtral 8x7B | 46.7B | 12.9B | 8 / 2 | 32K | The teaching example, still the clearest MoE to read code for |
| gpt-oss-120b | ~117B | ~5.1B | 128 / 4 | ~131K | Apache 2.0, MXFP4, fits one 80GB H100. The realistic self-host |
| Qwen3-235B-A22B | 235B | 22B | 128 / 8 | Long-context variants published | Multi-GPU. Active count is in the model name |
| DeepSeek-V3 | 671B | 37B | 256 routed + 1 shared / 8 | 128K | 8x H200 in vLLM's recipe. Frontier quality, data-centre bill |
| Newer frontier open weights | Industry reports suggest 700B to 1.6T | Reported in the 30B to 50B range | Often 256 / 8 | Up to 1M claimed | Figures circulate faster than they are confirmed. Verify on the model card |
The pattern to carry into an interview: as total parameters rise, the active fraction falls. Vendors are not adding compute per token, they are adding capacity for the router to choose from. Frontier closed models are widely assumed to work the same way, but no major lab publishes its layout, so treat anything specific about them as speculation.
What MoE Inference Cost Really Looks Like
Keep one scenario in your head, and treat it as illustrative rather than a real customer. A four-person AI team at a Pune logistics SaaS runs document extraction over consignment notes: roughly 12 million output tokens a month, bursty in business hours, nothing latency-critical because results land in a queue. They pay a frontier API today and want to know whether self-hosting an open-weight MoE is cheaper.
The arithmetic for DeepSeek-V3
vLLM's published recipe wants 8x H200 for the FP8 checkpoint. An October 2026 comparison of H200 cloud prices across providers listed on-demand rates starting near $2.09 per GPU-hour on the cheapest provider's own pricing page, several mid-market providers around $4.50, and Azure's 8-GPU H200 instance working out to roughly $10.60 per GPU. Take the cheapest credible figure: eight cards at $2.09 is about $16.70 an hour, close to $12,000 a month if the box stays up, and about $26,000 at the $4.50 rate. For 12 million tokens a month, that is a per-token cost no API would dare charge.
So the team should not self-host the 671B model. They should run gpt-oss-120b on a single 80GB card, which MXFP4 quantisation makes possible, and keep the frontier API for the 5% of documents the smaller model fumbles. One GPU instead of eight changes the decision completely, and the architecture is identical.
Cloud GPU rates move weekly and vary by region, so treat every figure above as a snapshot to re-check on the provider's pricing page. If costing GPU-backed infrastructure is the part you keep guessing at, that work sits inside the MLOps engineering track, and the rent-versus-reserve economics behind it is covered in the AWS architecture and DevOps programme.
Why the token price fell but the context price did not
When a provider cuts the per-token price of a large model, MoE sparsity is usually why: they bill against 37B of compute, not 671B. What does not fall is anything tied to memory. Long contexts, large KV caches and high concurrency still cost what they cost, which is why cheap tokens sit next to unchanged long-context surcharges.
Also read: vLLM and continuous batching, for why keeping the batch full is what decides whether any of this arithmetic works.
Learn why a 671B model runs at 37B, then build the systems that serve it
The AI Engineer course covers generative AI, RAG and AI agents across 100+ hours of live sessions over 16 weeks, certified on both Microsoft and Claude stacks. You build agents that plan, call tools and act, rather than only prompting a model.
Explore the course
What Usually Goes Wrong With MoE in Production
This part is not in the model card. Take one MoE model to production this year and you will hit at least two of these four.
Four failures that catch teams out
Each has a cheap tell you can check before committing hardware.
The VRAM surprise
Somebody reads "37B active" and provisions for a 37B model. The loader then asks for all 671B of weights and dies. Active parameters predict speed, never memory.
Size from the totalExpert imbalance under real traffic
Your traffic is narrower than the training mix, so the router keeps hitting the same experts. Across multiple GPUs, cards holding popular experts saturate while others idle, and throughput sits below the benchmark you were promised.
Watch per-GPU use, not averageServing at batch size 1
MoE economics come from filling the box. One concurrent request buys a 37B model's latency on a 671B model's hardware bill. If your p99 concurrency is one, you wanted a dense 32B.
Measure concurrency firstFine-tuning that breaks the router
A narrow domain dataset pushes the router toward a few experts and quietly degrades everything outside that domain. Keep a held-out general evaluation set and run it after every tuning job.
Evaluate outside your domainCompiled from vendor serving documentation and published MoE training literature, including the load-balancing discussion in arXiv 2412.19437. Checked 6 November 2026.
Mixture of experts is oversold as a democratisation story. It made frontier quality available in open weights, which is real, but it moved the binding constraint from compute to memory, and memory is exactly what a learner, a startup or a college lab cannot rent cheaply. A dense 8B on a laptop is still more useful to most people than a 671B MoE they will never load. If you are learning, learn the architecture and rent a small MoE by the hour. Do not buy hardware for this.
Is Mixture of Experts Worth Learning in 2026?
Yes, but not the way most roadmaps imply. You will almost certainly never train one; two kinds of organisation train MoE models at this scale and both have research budgets.
What you will do repeatedly is choose and serve them. That means reading a model card and knowing in thirty seconds whether it fits your hardware, explaining to a finance lead why a cheaper token price came with the same memory bill, and spotting that your throughput problem is expert imbalance rather than a slow GPU. Those judgements come up in system design rounds far more often than the mathematics of a gating function.
Where it shows up in certifications and job work
No current certification examines MoE routing directly. They examine the decision layer above it: model selection against cost and latency targets, deployment topology, evaluation. Microsoft's generative AI developer path runs through model catalogue and deployment choices, which is the practical home of this knowledge and what the Generative AI Developer course prepares for, while justifying a model choice to a client is the territory of CCAR-F architect foundations prep. Teams living in the Claude API usually meet this material through CCDV-F developer prep, and those building retrieval-heavy systems on Azure will recognise the same trade-offs from the Azure AI developer track. The full certifications overview lays the paths side by side.
Also read: small language models and SLM versus LLM costs, which is the opposite answer to the same question about where to spend compute.
How to Get Started With Mixture of Experts This Month
Four weekends, no hardware purchase, and you can defend a model choice in an interview.
A four-week route into MoE, from reading to serving
Each stage ends in something you can show or say out loud.
Read the two primary sources
Work through the Mixtral release notes and the architecture section of the DeepSeek-V3 technical report. Write down in your own words the difference between a shared and a routed expert.
Implement a toy router
Build a two-expert MoE layer in about 40 lines of PyTorch: linear gate, softmax, top-1 selection, weighted sum. Print which expert each token picked and watch imbalance appear on its own.
Serve a real MoE for one hour
Rent a single 80GB GPU, serve gpt-oss-120b with vLLM, and record tokens per second at concurrency 1, 8 and 32. The gap between those three numbers is the entire MoE argument, in your own data.
Write the one-page decision memo
For a workload you actually know, recommend MoE or dense with the VRAM figure, the concurrency figure and the monthly cost. That memo is the artefact to bring to an interview.
Put it inside a system
Wire the served model into a retrieval or agent pipeline so the model choice has measurable consequences. To do this with a cohort and a reviewer, start with a free webinar.
Hardware and model references as cited above, including the single 80GB card requirement for gpt-oss-120b from Nebius's model documentation. Checked 6 November 2026.
If you are the Pune team from earlier, the honest recommendation is the boring one. Run the 120B-class MoE on one rented card, keep the frontier API as the fallback for the hard 5%, and revisit self-hosting only when token volume is ten times what it is today. Buying eight H200s to serve 12 million tokens a month is a decision people make because the parameter count is impressive, not because the arithmetic works. Learn the architecture well enough to say that out loud in a planning meeting, then go build the pipeline around it: that systems work, not the routing mathematics, is what the AI Engineer course is built to take you through.
Related guides
- How Do Large Language Models Work? Tokens, Embeddings and Attention Explained for 2026 the layer underneath this one, worth reading first if attention is still fuzzy.
- How to Run an LLM Locally in 2026: Ollama Commands, Hardware and What It Really Costs the hands-on version of the memory constraint this guide keeps returning to.
- AI Engineer Salary in India 2026: Pay by Experience, City and the Skills That Move It what infrastructure judgement like this is actually worth in an offer.
- Machine Learning Engineer Salary in India 2026: Why the Same Title Pays 12 LPA and 35 LPA the serving and infrastructure skills that split that band.
- MLOps vs DevOps in 2026: 12 Criteria, Job Counts and Which One to Learn First in India which career track owns model serving once it reaches production.
- Cloud Engineer Jobs in Mumbai 2026: Pay Bands, the Data Centre Shift and Who Is Hiring where the GPU capacity these models need is actually being built in India.
Frequently asked questions
What is mixture of experts in an LLM, in simple terms?
It builds the feed-forward part of each transformer layer as many small networks instead of one big one, with a small router choosing which few to run per token. The model stores all of them but computes with a handful, so it behaves like a large model in quality and a small one in speed.
Does a mixture of experts model need less GPU memory?
No, and this is the most common misreading. Every expert must be resident in VRAM because the router can select any of them for the next token. DeepSeek-V3 activates 37B parameters per token but you provision for all 671B, roughly 671GB at FP8, which is why vLLM's published recipe lands on 8x H200.
What is the difference between active parameters and total parameters?
Total parameters are what you store and load. Active parameters are what multiply per token, so they set latency and compute cost. Qwen3-235B-A22B spells both into its name: 235B total, 22B active. A dense model has no gap between the two.
Is MoE better than a dense model?
Better at quality per GPU-hour when you can keep the batch full, which is why open-weight frontier models are nearly all MoE now. Worse when you serve one request at a time, have a single modest GPU, or need predictable latency. For a first deployment on limited hardware, a dense 8B to 32B is usually the right call.
Can I run a mixture of experts model on one GPU?
For the 120B class, yes. Nebius's documentation for gpt-oss-120b describes roughly 117B total parameters with about 5.1B active, quantised to MXFP4 so it fits a single 80GB H100. Anything from 235B to 671B needs multiple cards.
Is mixture of experts the same as routing between different models?
No. MoE routes a single token between experts inside one model, in every layer. Model routing sends a whole request to one of several separate models by difficulty or cost. Both are called routing, they solve different problems, and interviewers like the distinction.
Do I need to understand MoE to get an AI engineer job in India?
You need its consequences: how to read a model card, size hardware from total parameters, and explain a cost trade-off. Nobody will ask you to derive a gating function. Saying why a 671B model is cheaper per token but not per gigabyte is the level that matters in a system design round.
Which open-weight MoE model should I start with?
Read Mixtral 8x7B to understand the idea, since top-2 of 8 is easy to follow, and serve gpt-oss-120b when you want to feel the economics on real hardware. Leave the 671B models as case studies until your volume justifies the eight-GPU bill.
About this guide. 360 Digital Transformation is an Anthropic Partner and a Microsoft Authorized Training Partner. Other certification bodies, vendors and employers named here are not affiliated with us. Product features and pricing change often; figures cited were checked on 6 November 2026.
Python
Claude API
FAISS
ChromaDB
LangChain
LangGraph
CrewAI
AutoGen (AG2)
