Home › Guides › Model distillation
Tech Explained · 2026Model Distillation in 2026: How It Works, Distillation vs Fine-Tuning and the Real Cost Numbers
Model distillation trains a small student model to copy a larger teacher model's outputs, so you serve the cheap model in production and keep most of the quality. AWS's own function-calling write-up reports 72% lower latency moving from Llama 3.1 405B to a distilled Llama 3.2 3B. It only pays off at high request volume.
- The teacher labels, the student serves. Sample real prompts, have a large model answer them, train a small model on the pairs.
- Tokens are cheap, engineering is not. Labelling 50,000 examples runs about 100 dollars at reported list prices. The eval harness is the real bill.
- There is a volume floor. AWS guidance points at workloads above roughly 10 million invocations a month. Below that, setup eats the savings.
- Check the terms first. Anthropic's usage policy names model distillation as prohibited without prior authorisation.
- Most teams should not distil yet. A cheaper model plus a confidence router gets most of the saving in a fortnight.
Your support classifier works. It also costs about 2.5 lakh rupees a month in API calls, and last week finance asked for that number in writing. Every page you read says the same thing: train a small model on a big model's answers, save 90%. Nobody writes down the volume below which that loses money.
What Model Distillation Actually Does
Distillation moves behaviour, not weights. You take a large model that already does your task well, use it to answer thousands of your real inputs, and train a much smaller model to reproduce those answers. The student never sees the teacher's internals, only what it said. So a distilled 3B model will not match a 400B model in general, but it can match it on invoice classification for your five document types, because that is all it was asked to learn. You trade unused breadth for a bill you can afford.
The distillation loop, end to end
The eval gate is the step teams skip, and the one that decides this.
From the published Bedrock and Foundry distillation workflows, checked 10 November 2026.
How Knowledge Distillation Works in Practice
Textbook knowledge distillation trains the student on the teacher's full probability distribution over tokens, not just its final answer. That softer signal says not only that the right label was "billing dispute" but that "refund request" was a close second. Most managed platforms hand you the text only.
On Amazon Bedrock, the product page notes distillation can read prompts and responses straight from your invocation logs, so you skip paying the teacher to regenerate answers already on disk. In Microsoft Foundry the equivalent route was Stored Completions, and Microsoft's docs now mark it deprecated with a retirement date of 15 October 2026.
Teacher and Student Model: Who Plays Which Role
Pick the teacher on accuracy against your eval set, not on brand. Pick the student on what you can serve: parameter count, context window, and whether a quantized build fits the GPU you can rent. Microsoft's guidance accepted 10 stored completions but recommended hundreds to thousands. Ten is a floor the API tolerates, not one that yields a usable model.
Model Distillation vs Fine-Tuning vs Quantization: What Each One Buys You
Model distillation vs fine-tuning is the comparison people get wrong most often, so take the three apart. Fine-tuning adjusts an existing model's weights on your examples. Distillation produces a smaller model that imitates a bigger one, usually via fine-tuning. Quantization leaves behaviour alone and shrinks the arithmetic, so the same model fits cheaper hardware.
| Criterion | Model distillation | Fine-tuning (LoRA) | Quantization | Pick a cheaper model |
|---|---|---|---|---|
| Needs labelled data | No, the teacher labels | Yes, usually by hand | No | No |
| Setup time | 3 to 6 weeks | 1 to 3 weeks | Hours to days | Days of eval work |
| Main cost driver | Eval engineering, then serving | Labelling, then training | Negligible | None |
| Quality risk | Silent regression on tail cases | Overfitting | Small, measurable loss | Visible drop on hard inputs |
| Serving requirement | Own GPU or custom deployment | Adapter hosting | Own GPU | Stays managed |
| Reversible in a day | No, you own a model now | Partly, drop the adapter | Yes | Yes |
| Wins when | Very high volume, or data cannot leave | You need a specific format | You already self-host | Almost every first attempt |
Distillation vs Quantization: Different Problems
If you already self-host, quantization is the first thing to try, because it is a flag rather than a project. Distillation vs quantization is not really a choice: you distil to get a smaller model, then quantize that student.
Also read: LoRA Fine-Tuning in 2026: How Low-Rank Adapters Work, LoRA vs QLoRA, and Real GPU Costs.
What Model Distillation Costs in 2026
Three numbers decide this, and only one is the API bill.
The three numbers that decide a distillation project
The cheap one is the number everybody quotes.
First two from AWS's Bedrock material; the third is arithmetic on reported list prices. Checked 10 November 2026.
Be precise about what AWS measured. Its Bedrock machine learning blog reports that in function-calling tests, moving from Llama 3.1 405B to a distilled Llama 3.2 3B cut latency 72% and lifted output speed 140%, with the student close to the teacher's accuracy on BFCL v2 filtered to entertainment-domain calls. A filtered benchmark on a narrow task is what distillation is good at, and why the number will not transfer to your chatbot.
The Bedrock distillation page states the best case as up to 500% faster and up to 75% less expensive with minimal accuracy impact. Read it as a ceiling. The pricing page charges synthetic data generation at the teacher's on-demand rate, bills training separately, then requires a custom model on-demand deployment or Provisioned Throughput to serve the result. That last line turns a per-token bill into something nearer reserved capacity, and those economics are standard ground in 360DT's live AWS Solutions Architect and DevOps course, which runs the SAA-C03 track on real infrastructure.
The volume floor is published, not folklore. An AWS re:Post cost-optimization article says distillation suits high-volume repetitive workloads, treats workloads above 10 million invocations a month as its reference point, and warns that at low volume training cost exceeds inference savings.
Where the effort goes in a first distillation project
Training the model is the small part.
Illustrative split of engineer time, not measured data. Checked 10 November 2026.
Keeping an eval harness, registry and retrain schedule honest is model operations, not prompt work, and it is the spine of a live MLOps Engineer programme built on the AI-300 stack.
A Worked Example: 1.5 Million Classifications a Month
An illustrative scenario, not a real client: a two-person AI platform team at a mid-size Indian insurer, routing 1.5 million customer messages a month. Each call runs about 700 input tokens and 60 output tokens, so roughly 1,050 million input and 90 million output tokens a month.
Using the Claude list prices third-party trackers reported in early November 2026 (Sonnet 5.5 at 2 dollars per million input and 10 per million output, Haiku 5.5 at 0.10 and 0.50 under 100,000-token prompts, batch at half price), the arithmetic falls out as below. Confirm against the official pricing page before budgeting.
| Option | Monthly inference bill | Upfront investment | Quality risk |
|---|---|---|---|
| Mid-tier model, no optimisation | About $3,000 | None | Baseline |
| Same model via the batch API | About $1,500 | A day of queue plumbing | None, same model |
| Cheap hosted model, swapped in | About $150 | A week of eval work | Drops on the hardest slice |
| Cheap model plus confidence router | About $400 | Two weeks | Near baseline if tuned |
| Distilled open-weight student, self-hosted | GPU rental, not tokens | 3 to 6 weeks plus ~$100 of tokens | Silent drift without evals |
Read rows three and five together, because that is the real decision. The cheaper hosted model takes this team from 3,000 dollars to about 150 a month in roughly a week. Distillation might go lower on paper, but it costs six weeks of a two-person team, adds a GPU bill that does not sleep on a quiet weekend, and leaves them owning a model that needs retraining whenever the message mix shifts.
So the verdict here is: do not distil. Swap to the cheap model, build a router that escalates the uncertain 10% to the mid-tier model, and spend the rest of the quarter on the eval set. They land near 400 dollars a month, hold baseline quality where it matters, and keep both engineers free. 360DT's AI Engineer course builds that escalation path as live project work.
Distillation becomes right at one of three triggers: traffic crosses roughly 10 million calls a month, a product requirement sets a hard latency ceiling, or compliance rules that message text cannot leave their VPC. That last one is the most common real reason Indian BFSI teams distil, and it is not about cost.
Before you distil anything, learn the cheaper levers that beat it
The AI Engineer course teaches you to build agents that plan, use tools and act, certified on both Microsoft Copilot Studio and Claude Code. You practise retrieval, routing and evaluation on real builds, where a distillation project's promised savings usually come from.
Explore the course
Where Distillation Goes Wrong
Here is the one that gets people. Your student passes eval at 94% agreement with the teacher, you ship it, and nothing looks broken for six weeks. Then support starts mishandling a category that is 2% of volume and 40% of escalations, because it was 2% of your training sample too. Nobody gets paged for a confident wrong answer.
Four failures in every first distillation project
Each has a cheap fix if you plan for it.
Aggregate metrics hiding tail failures
One accuracy number will not show a collapsed minority class. Slice the eval set by category, language and length, then set a floor per slice.
Eval designTraining on the teacher's mistakes
Whatever the teacher gets wrong, the student learns as correct and defends confidently. Review a sample of teacher outputs before training, not after shipping.
Data qualityNo path back to the teacher
Ship the student behind a router that escalates on low confidence or a named category. No fallback turns every edge case into an incident.
ServingForgetting the retraining treadmill
Traffic drifts and products launch. Decide who owns the quarterly retrain, or you will serve a model tuned for last year's customers.
OwnershipPatterns drawn from the workflows and cost guidance cited above, checked 10 November 2026.
For most teams reading this, distillation is oversold. The first 70% of an inference bill usually comes off through a cheaper model, prompt caching, batching and a sane router, all four reversible in an afternoon. Distillation asks for a quarter of senior engineering time, hands you a model you own forever, and adds a retrain nobody budgeted. Fewer than three engineers and no eval infrastructure? Not your next move.
Designing the eval gate and the escalation boundary is architecture work, and it maps onto the agent design domains of the CCAR-F architect foundations exam; the certifications overview shows where each credential sits.
The Legal Line: You Cannot Distil Whatever You Like
Most technical write-ups skip this, and it is what will actually stop your project. Commentary in Lawfare notes that OpenAI, Meta and Anthropic have written prohibitions on distillation into their user agreements. Anthropic's usage policy names model scraping or model distillation as prohibited without prior authorisation, and Google's Gemini API terms bar using the service to develop competing models.
So your teacher choice is constrained by contract before benchmark scores come into it. Read the terms of whatever you planned to label with, and ask for written authorisation rather than assuming. The restriction is contractual rather than an intellectual property right, since model outputs are generally not copyrightable, and commentators note no court has comprehensively ruled on these clauses.
Open-weight teachers under permissive licences sidestep most of this, which is largely why published case studies use Llama and Qwen models. Check the specific licence, not the family reputation.
Is Model Distillation Worth Learning in 2026?
Yes, but second. The skills that get you shortlisted for AI engineering work in India are retrieval, agent design, evaluation and cost control. Distillation sits a layer behind them: what a senior engineer reaches for once the cheap levers are exhausted. Interviewers ask about it to see whether you know when not to use it.
Learn it in this order: build an eval set that slices by failure mode, serve an open-weight model yourself and measure cost per thousand requests, understand LoRA, then distillation as the technique combining all three. The AI-103 developer track covers the first two; teams on the Claude API will want the Agent SDK ground in the CCDV-F prep course.
Also read: What Is vLLM in 2026? How Continuous Batching Works, vLLM vs Ollama and Real GPU Costs per Million Tokens.
If I were the engineer at that insurer, I would spend this quarter making the eval set genuinely good and shipping the confidence router, then write a one-page note on why distillation is deferred and which trigger brings it back. That note is worth more at your next review than a distilled model nobody can prove is safe. The next step is 360DT's AI Engineer course, where the routing, retrieval and evaluation work that makes distillation unnecessary, and later survivable, is built live over 16 weeks.
Related guides
- What Are Small Language Models in 2026? How SLMs Work, SLM vs LLM Costs and 6 Steps to Get Started your student is usually an SLM.
- How Do Large Language Models Work? Tokens, Embeddings and Attention Explained for 2026 the token mechanics soft-label distillation needs.
- MLOps Projects in 2026: 6 Portfolio Builds With Real Pipelines, Registries and Drift Alerts build the plumbing before you need it.
- MLOps Engineer Salary in India 2026: Advertised Pay by Experience, Company Type and Skill Stack what these skills are advertised at.
Frequently asked questions
What is model distillation in simple terms?
Training a small model to imitate a large one. You collect real inputs from your app, have the large model answer them, then train the small model on those pairs. You serve the small model, and it matches the large one only on the task you trained it for.
Is model distillation the same as fine-tuning?
No, though distillation normally uses fine-tuning to do its work. Fine-tuning adjusts a model on examples you supply, whoever labelled them. Distillation is the case where a larger model generates the labels and the target is a smaller model.
How much data do you need to distil a model?
Microsoft's now-deprecated stored completions route accepted 10 examples but recommended hundreds to thousands. Treat the low number as an API limit, not guidance. For classification, plan on a few thousand examples that over-sample rare categories, because those are what a student loses first.
Is it legal to distil GPT or Claude?
Not without permission under the current terms. Anthropic's usage policy names model distillation as prohibited without prior authorisation, and legal commentary reports similar clauses from OpenAI, Meta and Google. The restriction is contractual and untested in court, but a breach is still a real risk. Open-weight teachers avoid most of it.
Distillation vs quantization: which should I do first?
Quantization, because it is a configuration change rather than a project and you can measure it in an afternoon. Distil only if a quantized smaller model is still not good enough. The two combine: distil, then quantize the student.
Do you need your own GPU to distil a model?
Not for training, if you use a managed path such as Amazon Bedrock's distillation feature. You do need somewhere to serve it: on Bedrock a custom model on-demand deployment or Provisioned Throughput, and self-hosting means renting a GPU continuously. Serving is where the recurring cost lives.
About this guide. 360 Digital Transformation is an Anthropic Partner and a Microsoft Authorized Training Partner. Other certification bodies, vendors and employers named here are not affiliated with us. Product features and pricing change often; figures cited were checked on 10 November 2026.
Python
Claude API
FAISS
ChromaDB
LangChain
LangGraph
CrewAI
AutoGen (AG2)
