# How Model Distillation Works: A Practical LLM Guide > Learn how model distillation works, from teacher-student training and soft targets to synthetic data, loss design, evaluation, and quantization tradeoffs. - Author: Swarnava Dutta (https://swarnava.dev) - Published: 2026-08-07 - Tags: AI Model Distillation, Model Distillation In Gen AI - Reading time: 10 min (2266 words) - Canonical: https://swarnava.dev/blogs/how-model-distillation-works --- ![Illustration of how model distillation works: A tall glass decanter labeled with layered liquid on the left slowly pours](/images/blogs/how-model-distillation-works-hero.jpg) The first time I tried to ship a distilled model, I made the classic rookie mistake: I trained the student on the teacher's final answers only, like it was just another supervised fine-tune. Two weeks later, the student was confidently wrong in ways the teacher never was - hallucinating citations, botching multi-step math, missing the reasoning the teacher clearly "knew" but never wrote down explicitly. I burned a weekend digging through logits before I understood why. That's the gap most people miss when they ask how model distillation works. It's not just "train a small model to copy a big one's outputs." The real trick is transferring the teacher's full probability distribution - its uncertainty, its runner-up guesses, the shape of its confidence - not just the single token it picked. This guide walks through the actual teacher-student pipeline: building transfer data, generating soft targets, choosing loss functions, and evaluating whether your student actually learned something or just memorized a smaller slice of the same mistakes. I'll also cover where distillation earns its keep versus quantization, and the failure modes that cost me time I'd rather you skip. ## How Model Distillation Works in a Teacher - Student Setup Every distillation setup has three moving parts: a teacher model, a student model, and a transfer dataset that connects them. The teacher is usually a large, expensive-to-run model - think a frontier-scale LLM - that already performs well on the target task. The student is a smaller architecture you actually want to deploy, and the transfer dataset is the set of prompts and teacher outputs the student trains against. The core move isn't copying weights - it's copying *behavior*. You're not cloning the teacher's parameters into a smaller shape; you're teaching a differently-sized network to approximate the same input-output mapping, sometimes even the same internal representations if you're doing feature-level distillation. That's why soft targets matter so much. A one-hot label tells the student "the answer is token X," full stop. A soft probability distribution tells the student "X is most likely, but Y and Z are plausible too, and here's roughly how plausible." That extra structure is where distillation earns its name - the teacher is quietly encoding which wrong answers are *almost right*, and which are nonsense. This is also the answer to why model distillation works at all: the teacher smooths the learning target, exposing similarities among classes that a single label erases entirely, similar in spirit to how [large language models](/blogs/how-large-language-models-work) build internal representations during pretraining. ## How AI Model Distillation Is Done: The End-to-End Pipeline Distillation in Gen AI splits into offline and online flavors. Offline distillation generates and stores teacher outputs once, then trains the student against that frozen dataset - cheaper, reproducible, easy to audit. Online distillation queries the teacher live during student training, which adapts better but chains your training loop to teacher inference cost and latency. Most production pipelines start offline. You need storage for prompts and teacher responses, a training cluster for the student, and experiment tracking that logs which dataset version, temperature, and loss weights produced which checkpoint - skip that logging and you'll lose a week reconstructing which run actually worked. ![Flow diagram showing model distillation pipeline from defining target capabilities through transfer dataset, teacher generation, student training, to deployment and monitoring](/images/blogs/how-model-distillation-works-diagram-1.jpg "The end-to-end distillation pipeline") ### Build a Representative Transfer Dataset Sample prompts that mirror real production traffic, not just benchmark-style questions. Cover task types, difficulty levels, domains, and languages in the same rough proportions your users will hit, and deliberately oversample edge cases the teacher handles well but the student might not. Split into train, validation, and a contamination-resistant test set before generating any teacher outputs - decide the split first, generate second. Strip duplicates, unsafe content, and low-quality prompts before they ever reach the teacher. ### Generate Soft Targets and Synthetic Responses Capture teacher logits when the API exposes them; raise temperature slightly to surface the runner-up tokens the teacher considers plausible. When logits aren't available, fall back to sequence-level supervision using the teacher's generated text, rationales, or tool-call traces as the target. Sample the teacher multiple times per prompt for harder tasks, then filter for correctness, format compliance, and consistency before training. ### Train and Iterate on the Student Model Initialize a student sized for your latency budget, tokenize consistently with the teacher's outputs, and blend synthetic supervision with real ground-truth data rather than trusting synthetic data alone. Tune temperature, loss weighting, and sequence length against validation performance, then regenerate weak slices of data as gaps show up. ## Distillation Loss Functions and Why They Work The standard distillation objective blends two terms: cross-entropy against the hard label, and a divergence term measuring how far the student's soft predictions sit from the teacher's. You weight them with a scalar, usually favoring the soft term once training stabilizes. KL divergence is the usual choice for the soft-target loss. You scale both teacher and student logits by a temperature before the softmax, which flattens the distributions and exposes the smaller probabilities on runner-up tokens - that flattening is the whole point, since a sharp distribution barely differs from a hard label. Beyond response-level matching, you can distill at different depths. Logit-level distillation matches output distributions token by token; feature-level distillation matches intermediate hidden states or attention patterns, similar to how [transformer attention mechanisms](/blogs/transformer-attention-mechanism-explained) build layer-wise representations; relation-level distillation matches the relationships between examples rather than individual outputs. When you're stuck with an API-only teacher - no logits, no hidden states - you fall back to sequence-level distillation: train on the teacher's generated text as a target, treating the whole response as the label. It works, but it's a blunter instrument. The real tension is imitation versus generalization. Distill too tightly and the student inherits the teacher's blind spots and stylistic quirks; distill too loosely and you lose the transfer benefit you paid for. ## A Practical Model Distillation Example for a Smaller LLM Say you're running a large general-purpose LLM as a customer support assistant and it's costing too much per ticket to justify the latency. The goal: distill it into a smaller domain-specific model that only needs to handle billing, shipping, and account questions, with a response budget under a second. Start by pulling a year of real support transcripts, filtering for the target categories, and hand-verifying a subset of answers as ground truth. Feed the rest through the teacher with controlled decoding - moderate temperature, a few samples per prompt - then run automated filters for tone, factual consistency, and format before anything touches training data. Fine-tune a small student checkpoint on that mix of verified human answers and filtered synthetic responses, adding a logit-matching loss if the teacher API exposes token probabilities. Compare four things side by side: the student's base checkpoint, the teacher, a plain fine-tune with no teacher supervision, and your distilled model. The expected outcome is a student that answers most billing and shipping questions with teacher-level correctness, runs far cheaper and faster to serve, and shows a measurable quality gap only on the rare edge cases the teacher handles that never showed up in your transfer data. ## Evaluate Quality, Efficiency, and Distillation Failures Ship-readiness isn't one number. You need task accuracy, instruction-following fidelity, calibration, hallucination rate, safety behavior, and out-of-distribution robustness measured separately, because a student can nail accuracy while its confidence scores go haywire. On the efficiency side, benchmark latency, throughput, memory footprint, checkpoint size, and per-request cost on the actual hardware you'll deploy to - a GPU-bound eval that ignores your CPU inference box tells you nothing useful. Pair generic benchmarks with human evaluation and task-specific tests; a model can top a leaderboard and still fumble your actual support tickets. Explicitly test whether the student inherited the teacher's biases, hallucination patterns, unsafe outputs, or stylistic tics - distillation transfers flaws just as efficiently as skills. Watch for a handful of recurring failure modes: - Narrow synthetic data that trains the student for a benchmark, not the job - Low-diversity teacher sampling that collapses the student's output range - Capability cliffs where the student handles easy cases fine but falls off a cliff on harder ones - Benchmark leakage from transfer data overlapping your eval set - Overfitting to the teacher's exact phrasing instead of the underlying task Set acceptance thresholds and regression tests before deployment, not after - decide what "good enough" means while you can still retrain cheaply. ## Model Distillation vs Quantization: Which Should You Use? Quantization doesn't train anything new - it takes an existing model's weights and shrinks their numerical precision, from FP16 down to INT8 or INT4, keeping the same architecture and same learned behavior. Distillation trains an entirely different, smaller network from scratch against teacher supervision, which costs more upfront but gives you freedom to change architecture, layer count, and vocabulary. Quantization is the faster operational choice: apply it in an afternoon, no transfer dataset, no training loop, often just a calibration pass over sample inputs. Distillation demands data generation, training infrastructure, and iteration cycles measured in days or weeks, but it buys real architectural flexibility - you can go from a dense model to something smaller in every dimension, not just lower-precision. ![Side by side comparison of model distillation training a new student model versus quantization reducing numerical precision of an existing model](/images/blogs/how-model-distillation-works-diagram-2.jpg "Distillation vs quantization: two paths to efficiency") Memory savings differ too: quantization roughly halves footprint per precision step, while distillation's savings depend entirely on how much smaller you make the student. Quality loss from quantization is usually mild and predictable; distillation's quality loss shows up unevenly, worse on tasks underrepresented in your transfer data. The two stack well: distill first, then quantize the resulting student for cumulative gains - smaller architecture and lower precision together. Pruning and low-rank adaptation are separate levers again - pruning removes weights or neurons from an existing model, LoRA adds small trainable adapters rather than retraining the whole network. ## Model Distillation Best Practices and Deployment Guardrails Before you shrink anything, write down the constraint that justifies the project: a latency ceiling, a cost-per-request cap, a specific capability the student must hit. "Make it smaller" isn't a target; "under 300ms at 90% of teacher accuracy on billing intents" is. Keep your verified, human-authored examples as the backbone of training and treat synthetic teacher output as augmentation, not the whole diet. Log everything that touched the pipeline - teacher version, prompt templates, decoding temperature, filtering rules, dataset licenses - because six months later you will need to explain why a checkpoint behaves the way it does. Read the terms of service before you distill from a third-party API; some vendors explicitly restrict using their outputs to train competing models, and that risk sits outside your training code entirely - I've covered the [legal exposure of distillation](/blogs/is-model-distillation-legal) in more depth elsewhere. Run privacy, safety, and bias review passes on both the transfer data and the final student, since flaws propagate quietly through soft targets. In production, monitor for drift and route uncertain or high-stakes requests to a larger fallback model rather than trusting the student everywhere. And skip distillation entirely when the task shifts weekly, the teacher itself is mediocre, your eval data is thin, or traffic volume doesn't justify the engineering cost. ## FAQ ### How model distillation works A large teacher model generates outputs - soft probability distributions, generated text, or both - on a set of prompts, and a smaller student model trains to match those outputs rather than just hard labels. The soft targets carry information about which wrong answers are "almost right," which gives the student a richer signal than standard supervised training. Done well, the student ends up approximating the teacher's behavior at a fraction of the size and cost. ### How is model distillation done In practice it's a pipeline: build a transfer dataset representative of production traffic, run it through the teacher to capture logits or generated responses, then train the student against a blended loss of hard labels and teacher soft targets. You iterate - regenerating weak data slices, tuning temperature and loss weights - until validation metrics close the gap with the teacher. Offline pipelines dominate because they're cheaper and easier to audit than querying the teacher live. ### How to model distillation Start with a clear constraint (latency, cost, or capability target), pick a teacher that already performs well on your task, and generate a transfer dataset that mirrors real usage rather than benchmark questions. Train a student sized to your budget using a combined cross-entropy and KL-divergence loss, then evaluate on accuracy, calibration, and efficiency before shipping. ### How does AI model distillation work It works because a teacher's probability distribution encodes more than a single correct answer - it encodes relative confidence across plausible alternatives. Training the student to match that distribution, rather than a one-hot label, transfers a smoothed, more informative version of what the teacher learned, which is why distilled students often generalize better than models trained from scratch on the same data volume. ## Further Reading 1. [Triplet Loss for Knowledge Distillation](https://arxiv.org/abs/2004.08116v1) - Hideki Oki, Motoshi Abe, Junichi Miyao et al. (2020) 2. [Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10](https://arxiv.org/abs/2605.31191v2) - Umut Onur Yasar (2026) 3. [SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization](https://arxiv.org/abs/2411.10958v7) - Jintao Zhang, Haofeng Huang, Pengle Zhang et al. (2024)