What is model distillation? It is a way to train a small AI model by having it imitate a larger one. The large model, called the teacher, produces outputs; the small model, called the student, learns to reproduce them. The goal is a model that is cheaper and faster to run but keeps much of the teacher’s skill. The technique is old and widely used, and in 2026 it is also at the center of a dispute over copying frontier AI models without permission.

This guide explains how distillation works, what it buys, and where the line is drawn between ordinary use and what AI companies now call adversarial distillation, using the original research and the companies’ own statements.

Quick answer Distillation trains a student model to match a teacher model’s outputs instead of learning only from raw data. Done on your own models, it is a standard way to shrink a system for deployment. Done on someone else’s model without authorization, it is what OpenAI and Anthropic describe as an attack, because the copy may lack the original’s safeguards.

Where the idea comes from

The method most people cite is the 2015 paper Distilling the Knowledge in a Neural Network by Geoffrey Hinton, Oriol Vinyals and Jeff Dean. They started from a practical problem: a very simple way to improve almost any machine learning system is to train many models and average their predictions, but running a whole ensemble is too expensive to deploy to many users. The paper builds on earlier work showing that the knowledge in an ensemble can be compressed into a single model that is much easier to deploy.

The paper’s key move is in how the small model is trained. Instead of learning only from the correct answers, it is trained on the class probabilities the large “cumbersome” model produces, which the authors call soft targets. Those probabilities carry information the correct label alone does not: the paper’s example is that an image of a BMW may have only a very small chance of being mistaken for a garbage truck, but that is still many times more probable than mistaking it for a carrot, and these relative probabilities show how the large model tends to generalize. To make the probabilities informative, the authors raise the temperature of the final softmax until the large model produces a suitably soft set of targets, then use the same temperature when training the small model. They report that distilling an ensemble into a single model significantly improved the acoustic model of a heavily used commercial system.

How the teacher and student setup works

In plain terms, distillation has three steps:

  1. Train or choose a teacher. This is a large, capable model, or an ensemble of models.
  2. Collect the teacher’s outputs. Depending on the setup, that can be probabilities over possible answers, generated text, step-by-step reasoning, or the teacher’s judgments about which answers are better. OpenAI describes the range this way: a model can learn from another model’s answers, its reasoning, or its judgments about which answers are better and why.
  3. Train the student to match them. The student is smaller, so it is cheaper to store and run, and it learns from the teacher’s behavior rather than building everything from scratch.

Because the student learns from the teacher’s outputs, how much it can learn depends on how much of the teacher it can see. That matters later in this guide.

What distillation buys: smaller, faster models

The clearest example in the research record is DistilBERT. In their 2019 paper, Victor Sanh and colleagues used distillation while pre-training a smaller version of the language model BERT. They report that it cut the size of the model by 40% while keeping 97% of its language understanding abilities and running 60% faster, and they demonstrated it in an on-device test. Those figures are the authors’ own results for that model and those benchmarks, not a general rule for every distilled system.

That trade is the reason distillation shows up wherever models must run on limited hardware. It is one route to the compact systems discussed in our guide to small language models versus large ones, and it starts from a teacher like the large language models most people know.

When distillation becomes an attack

Distilling a company’s own model is ordinary engineering. The dispute is about doing it to someone else’s model through its public interface.

OpenAI’s November 20, 2026 post on combatting adversarial distillation describes it this way: an actor may use a frontier model’s outputs without authorization to train another model with similar capabilities, then deploy it without comparable protections. OpenAI says that, beyond intellectual property concerns, the safety risk is that a second model can learn advanced capabilities without inheriting the safeguards that govern how the original may be used. Its example is cyber capability: a distilled model could help carry out attacks the original would have refused.

OpenAI puts particular weight on a model’s protected reasoning, the intermediate steps it generates while working toward an answer, because those traces can contain information the final answer leaves out. In an internal study, OpenAI says a student trained on exact reasoning traces generally outperformed one trained on reasoning summaries after supervised fine-tuning, and that protecting traces can limit how much cyber capability transfers. It also reports that, in an internal red-teaming evaluation, the average attack success rate for recovering protected reasoning fell from 54.0% against GPT-6 Astra to 0.2% against GPT-6.2 Astra. These are company-reported results.

Anthropic’s October 12, 2026 post on defenses against distillation attacks describes such attacks as covert, industrial-scale campaigns, often run through networks of fraudulent accounts, that generate millions of responses from a larger model to teach a smaller one. It says its methods reduced the data harvested by distillers by around 90 to 99% according to early evidence, and acknowledges trade-offs: some developers had to update integrations, and its classifiers can in fewer than 0.1% of cases misidentify or block benign requests. Both companies are describing their own defenses and their own findings, and neither claim has been independently verified here.

The same idea reached the daily news this week; see Daily 9 for November 21, 2026.

What to take from it

  • Distillation is a technique, not a verdict. Compressing your own model is routine engineering; doing it to another company’s model without authorization is what OpenAI and Anthropic call an attack.
  • The student can only learn what it can observe. That is why AI companies focus on hiding or protecting reasoning traces rather than only final answers.
  • Safeguards do not transfer automatically. A student trained on a teacher’s outputs learns behavior; OpenAI’s concern is that it may not inherit the rules around that behavior.
  • Treat company figures as claims. Attack success rates, percentages of harvested data and benchmark retention come from the organizations that built or defend the systems.