知识蒸馏：大模型如何教会小模型变聪明

人工智能已经融入你的工程工作流程。虽然代币消耗体现了这一点，但吞吐量却没有。人类仍然深度参与其中，这就是一个上下文问题。

8月19日加入直播（免费）：https://getunblocked.link/NwybmZ3 来学习：

在生产之前测量人工智能收益流失的四个指标。

上下文成熟度的八个阶段，你的指标受限的具体障碍，以及一个定位你团队所在阶段的免费工具。

为什么更多的MCP和更大的上下文窗口不足以保证价值，以及如何从你的智能体获得真正的价值。

立即注册：https://getunblocked.link/NwybmZ3

最强大的人工智能模型也是运行成本最高的。它们需要专用硬件，占用大量内存，并增加每个请求的成本和延迟。

这些特性使它们难以在资源有限的地方部署，例如移动设备或需要快速低成本响应的高流量服务。

还有第二个一开始听起来有些违反直觉的事实。小型模型有时可以在特定任务上匹配甚至超过大型模型，即便小型模型的知识完全来自大型模型。从直觉上看，一个从另一个模型输出中训练的模型似乎只会继承上限而无法突破，但结果却不同。实现这一点的方法称为知识蒸馏，它已成为构建生产级人工智能系统的标准部分。

在本文中，我们将从基础介绍这一概念。我们将讨论的主要内容如下：

蒸馏是什么，它与压缩有何不同。

为什么从模型输出中学习可能胜过从原始标签学习。

三种主要方法，以及哪种方法占主导。

蒸馏模型在实际中能够实现的效果。

该方法的局限性所在，以及未来的发展方向。

蒸馏训练一个新的、小型模型以复制大型模型的行为。该设置涉及两个模型：

第一个是名为教师的大型能力模型。

第二个是名为学生的小型模型，它被训练以复制教师的输出。

一旦训练完成，学生模型将独立运行，而教师模型则退居幕后。

一个常见的假设是学生模型是压缩形式的教师模型。然而，现实情况却不同。

诸如量化和剪枝之类的压缩方法是从一个模型出发，通过以较低精度存储数值或移除对结果贡献较小的部分，来减少其占用空间。模型本身仍然是原来的模型，只是更小、更轻。

而蒸馏则生成一个真正独立的模型，拥有自己的参数，通常设计也不同，其训练目标是在训练过程中表现得像教师模型。

一个操作是缩小现有模型，另一个是训练一个新的模型。实际意义明显，因为小型学生模型可以运行在单个服务或手机上，响应时间更短，每次请求成本更低，在某些情况下，它可以在设备本身运行而无需将数据发送到其他地方。

这种方法现在已成为标准做法。例如，谷歌的Gemma模型在训练过程中使用蒸馏，借鉴Gemini系列中的大型模型。这两种思路也可以顺序结合使用。模型通常先经过蒸馏生成一个较小但有能力的模型，然后再进行量化，以进一步缩小模型以适配特定设备。

保持这种区分很重要，因为它会影响我们对后续概念的理解。压缩模型内部包含原始模型的一份拷贝。蒸馏模型则是独立的，它经过训练以表现得像原始模型，这也正是为什么它有时会表现出原始模型无法表现的行为。

如果学生模型只是复制教师模型，为什么复制效果会如此好？

答案在于教师模型传递的内容。

从模型输出中学习优于从原始数据中学习，因为输出比单纯的答案携带了更多信息。

标准训练数据为每个样本提供一个答案。一张猫的图片带有“猫”的标签，模型若输出“猫”会获得奖励，而输出其他会受到惩罚。教师模型则提供更丰富的信息。它的输出不是单一答案，而是一组对各选项的概率，例如猫为0.70，狗为0.25，狐狸为0.05。这整组概率被称为软标签，与普通数据中的单一硬标签相对。

这些额外的数字携带了额外信息。它们显示教师模型将狗视为一种可接受的替代，而狐狸则是远离的选择，这说明了不同类别之间的关系。研究人员有时称之为暗知识，意思是模型信心中隐藏的结构，而单纯的标签未能体现。

在训练过程中，学生模型努力匹配这种分布。其得分取决于自身概率与教师概率的距离，训练会推动它缩小差距。换句话说，学生学习的是教师的整体信心模式，而不仅仅是单一正确答案，这种模式比一个词的标签提供了更强的训练信号。

这就是蒸馏方法之所以有效的核心原因。单一正确标签丢弃了选项之间的关系，而软标签则保留了这些关系。

早期的结果显示了实际效果，因为在软目标下训练时，学生模型能用更少的样本达到不错的性能，因为每个样本现在承载的不只是一个答案。2015年的原始工作增加了一个叫温度的控制器，正是为了这个目的，更高的温度可以将概率分布展开，使学生可以学习到更多的细微结构。

机制清楚之后，下一个问题是如何在实践中完成，这又有不止一种答案。

蒸馏有三种主要形式，而它们区别在于学生复制的内容不同：

输出蒸馏：学生匹配教师的最终输出，包括上文提到的软标签。这是2015年的原始形式，也是最直接的一种。

特征蒸馏：学生模型匹配教师模型的内部表示，即模型在处理输入时计算的中间值，然后才得出最终答案。目标是获得相似的内部图景，而不仅仅是相似的输出。谷歌的 EmbeddingGemma 就是以这种方式训练的，学会产生接近更大 Gemini 模型的内部表示。

合成数据蒸馏：教师模型生成一组示例数据集，学生模型以与训练普通数据相同的方式在该数据集上进行微调。斯坦福的 Alpaca 是早期案例之一，它在现有大模型生成的示例上进行微调，以提高遵循指令的能力。

第三种形式已成为实践中最常见的方法，部分原因与访问权限有关。

许多强大的模型只能通过返回文本的界面访问，其内部值和概率保持私密。当这些内部信息无法获取时，生成数据仍然是可行的途径。

这三种形式在所需条件上也有所不同。输出蒸馏需要教师模型的概率，特征蒸馏需要访问其内部值，而合成数据蒸馏只需要教师模型生成的文本，这也是它在封闭模型中传播最广的原因。

这些方法也可以结合使用。一次训练过程可能同时使用生成的数据集和软标签，而较新的方法在训练过程中混合教师模型和学生模型的生成。

这些方法不仅是理论上的。下一节展示了它们的产出。

实践中的结果表现强劲，但有一个重要的限定条件。

一个明确的例子出现在 2025 年初，由一个名为 DeepSeek 的实验室完成。它使用一个大型推理模型生成一组训练示例，然后在这些示例上微调几个现有的小模型。其中一个结果尤其突出。

一个拥有70亿参数的学生模型在一项竞赛数学基准测试中得分高于一个拥有320亿参数的模型，尽管它仅是通过对大型模型输出的简单微调生成的。发布的蒸馏模型家族参数从15亿到700亿不等，其中较小的模型足够紧凑，可以在单张显卡上运行，这也是该发布引起广泛关注的原因之一。其实际效果是，在这些狭窄任务上的出色表现变成了一个小团队可以本地、低成本运行的东西，而不仅仅依赖大型托管模型。

限定语和标题本身同样重要。

这些胜利往往出现在狭窄、定义明确的任务上，比如数学和编码。在这些任务上，小型蒸馏模型的表现可以超过其规模所暗示的水平。在更广泛的一般知识衡量标准上，同样的小模型仍然落后于大模型。例如，一个模型可以通过蒸馏在竞赛数学上变得非常出色，同时在关于世界的广泛问题上仍然较弱。因此，小模型击败大模型的说法通常只在特定、狭义的情况下成立。

如果结果如此出色，自然而然的问题是该方法的不足之处，而下一节将直接探讨这个问题。

蒸馏有明确的局限性，这在决定其是否适合特定问题时非常重要。

教师的天花板效应：基于教师输出训练的学生模型往往在其见过的数据类型上保持在教师水平或以下。当教师给出错误答案时，学生会连同正确答案一起学习这个错误答案。教师的质量设定了标准，这使得教师的选择成为整个过程中特别关键的决策之一。

差距过大可能有害：更大、更强的教师不一定能造就更好的学生。当教师和学生之间的差距非常大时，迁移效果可能会下降，因为学生的能力不足以吸收大模型表达的一切内容。关于这种能力差距的研究发现，最强的可用教师有时并不是最佳选择。存在一系列方法通过增加一个中间步骤来弥合较大的差距，即教师先训练一个中等规模的模型，然后该模型再训练小规模学生，这样每次传递的距离就较小。

架构可能比规模更重要：基础模型的设计可能比参数数量更关键。在一项研究中，一个32亿参数的学生在同一任务上超过了一个70亿参数的学生，因为较小的模型采用了更强的基础架构。仅凭规模很难预测蒸馏的效果。

教师可以传递的不仅仅是任务：在2025年发表在《自然》杂志上的一项研究中，具有某种特质（偏爱猫头鹰倾向）的教师模型被用于生成仅由数字序列组成的训练数据。学生在这些数字上训练后也表现出相同的猫头鹰偏好，即使数据被过滤以移除任何可见特质痕迹。类似效果也出现在更严重的行为上，并且仅发生在教师和学生共享相同基础模型时。结论是，蒸馏可以传递的不仅仅是所教授的任务，有时仅仅过滤可见数据不足以阻止它。

这些限制设定了边界，在边界内，方法持续进步。下一节介绍它的发展方向。

蒸馏的最新方向是通过自动化整个过程来减少人工工作量。

在这种设置中，大模型独立运行完整循环。它生成训练数据，对学生进行微调，用自己生成的一组保留样本来评估学生的表现，并重复循环，调整生成内容，直到学生不再进步。人的角色缩小为开始时定义任务和成功标准，并在最后对真实数据进行最终检查。

2026年的最新研究将此方法应用于检测任务，发现效果很好，其中一个值得记住的发现值得关注。

教师模型的选择对结果有很大影响。不同的教师模型，在相同的循环和相同的学生模型下，会产生明显不同质量的学生模型。因此，自动化减少了人工操作，但也使初始教师的选择更为关键，因为这一选择现在将驱动整个自运行过程，而不仅仅是一次训练。

同样的循环也显示了随着时间推移，手工构建的管道工作会减少，因为更多的数据生成和评估由模型自身完成。对于一个团队来说，吸引人的地方在于可以构建一个小型、任务特定的模型，而无需先组建大型手工标注的数据集，因为教师模型提供了训练样本和用于评分的数据。

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live on Aug 19 (FREE) ：https://getunblocked.link/NwybmZ3 to learn:

The 4 metrics to measure where AI gains leak out before production.

The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is

Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.

Register now ：https://getunblocked.link/NwybmZ3

The most capable AI models are also the most expensive to run. They need specialized hardware, they consume large amounts of memory, and they add cost and delay to every request they handle.

These traits make them hard to deploy in places where resources are limited, such as a mobile device or a service that handles heavy traffic and needs fast and low-cost responses.

There is also a second fact that sounds backward at first. A small model can sometimes match or beat a much larger model on a specific task, even when the small model learned everything it knows from the larger one. On an intuitive level, a model trained on another model’s output would seem to inherit a ceiling rather than break through it, yet the results are different. The method that makes this work is called knowledge distillation, and it has become a standard part of how production AI systems get built.

In this article, we will walk through the idea from the ground up. The main points we will cover are as follows:

What distillation is, and how it differs from compression.

Why learning from a model’s output can beat learning from raw labels.

The three main methods, and which one dominates?

What distilled models achieve in practice.

Where the method breaks down, and where it is heading next.

Distillation trains a new, smaller model to copy the behavior of a larger one. The setup involves two models:

The first is a large, capable model called the teacher.

The second is a smaller model called the student, which is trained to reproduce the teacher’s outputs.

Once training finishes, the student runs on its own, and the teacher steps out of the picture.

A common assumption is that the student is the teacher in compressed form. The reality, however, is different.

Compression methods such as quantization and pruning start with one model and reduce its footprint by storing its numbers at lower precision or removing parts that contribute little to the result. The model stays the same model, smaller and lighter.

Distillation, on the other hand, produces a genuinely separate model, with its own parameters and often a different design, whose goal during training is to behave like the teacher.

One operation shrinks an existing model. The other trains a fresh one. The payoff is practical, since a small student can run inside a single service or on a phone, respond in less time, and cost far less for each request, and in some cases, it can run on the device itself without sending data elsewhere.

This method is now standard practice. For example, Google’s Gemma models are built using distillation during training, drawing on a larger model in the Gemini family. The two ideas also work together in sequence. A model is often distilled first to produce a smaller capable model, then quantized to shrink that model further for a specific device.

Keeping the distinction clear matters because it affects how we understand further concepts. A compressed model carries a copy of the original inside it. A distilled model is a separate thing that was trained to act like the original, which is exactly why it can sometimes behave in ways the original would not.

If the student only copies the teacher, why does copying work so well?

The answer is in what the teacher hands over.

Learning from a model’s output beats learning from raw data because the output carries more information than a plain answer.

Standard training data gives one answer per example. An image of a cat carries the label “cat,” and the model is rewarded for producing “cat” and penalized for anything else. A teacher model offers something richer. Instead of a single answer, its output is a set of probabilities across the options, such as cat at 0.70, dog at 0.25, and fox at 0.05. That full set of probabilities is called a soft label, in contrast to the single hard label found in ordinary data.

The extra numbers carry additional information. They show that the teacher’s output ranks dog as a plausible alternative and fox as a distant one, which says something about how the categories relate to each other. Researchers sometimes call this dark knowledge, meaning the structure hidden in a model’s confidence that a bare label leaves out.

During training, the student works to match this distribution. It is scored on how far its own probabilities sit from the teacher’s, and training pushes it to close that gap. In other words, the student learns the teacher’s whole pattern of confidence rather than a single right answer, and that pattern is a stronger training signal than a one-word label.

This is the core reason distillation works as well as it does. A single correct label discards the relationships between options, and soft labels keep them.

An early result showed the practical payoff, since a student could reach good performance from far fewer examples when trained on soft targets, because each example now carried more than a single answer. The original 2015 work added a control called temperature for exactly this purpose, where a higher temperature spreads the probabilities out and exposes more of that fine structure for the student to learn.

With the mechanism clear, the next question is how this gets done in practice, which has more than one answer.

Distillation comes in three main forms, and they differ in what the student copies:

Output distillation: The student matches the teacher’s final outputs, including the soft labels described above. This is the original form from 2015 and the most direct one.

Feature distillation: The student matches the teacher’s internal representations, meaning the intermediate values a model computes while processing an input, before it settles on a final answer. The aim is a similar internal picture, not only a similar output. Google’s EmbeddingGemma is trained this way, learning to produce internal representations close to those of a larger Gemini model.

Synthetic data distillation: The teacher generates a dataset of examples, and the student is fine-tuned on that dataset the same way it would be trained on any ordinary data. Stanford’s Alpaca was an early case, fine-tuned on examples produced by an existing large model to improve how well it followed instructions.

The third form has become the most common approach in practice, and part of the reason comes down to access.

Many strong models are reachable only through an interface that returns text, with their internal values and probabilities kept private. When those internals are out of reach, generating data is the route that still works.

The three forms also differ in what they require. Output distillation needs the teacher’s probabilities, feature distillation needs access to its internal values, and synthetic data distillation needs only the text the teacher produces, which is why it travels the furthest across closed models.

These methods can also be combined. A single training run might use a generated dataset alongside soft labels, and newer methods mix teacher and student generation during training.

These methods are not only theoretical. The next section shows what they produce.

The results in practice are strong, with one important qualifier.

A clear example came in early 2025 from a lab called DeepSeek. It used a large reasoning model to generate a set of training examples, then fine-tuned several existing smaller models on those examples. One result stood out.

A 7-billion-parameter student scored higher than a 32-billion-parameter model on a competition mathematics benchmark, even though it was produced by plain fine-tuning on the larger model’s outputs. The released family of distilled models ran from 1.5 billion parameters up to 70 billion, and the smaller ones were compact enough to run on a single graphics card, which is part of why the release drew so much attention. The practical effect was that strong performance on these narrow tasks became something a small team could run locally and cheaply, rather than only through a large hosted model.

The qualifier matters as much as the headline.

These wins tend to appear on narrow, well-defined tasks such as mathematics and code. On those tasks, a small distilled model can perform at a level its size would not suggest. Across broader measures of general knowledge, the same small models still trail the larger ones. For example, a model can become excellent at competition mathematics through distillation while remaining weaker at wide-ranging questions about the world. Therefore, a claim that a small model beats a large one is usually true in a specific, narrow sense.

If the results are this good, the natural question is where the method falls short, which the next section takes on directly.

Distillation has clear limits, and they matter when deciding whether it fits a given problem.

A ceiling effect from the teacher: A student trained on a teacher’s output tends to stay at or below the teacher’s level on the kind of data they saw. When the teacher produces a wrong answer, the student learns that wrong answer along with the right ones. The teacher’s quality sets the bar, which makes the choice of teacher one of the most consequential decisions in the process.

A wider gap can hurt: A larger, stronger teacher does not always produce a better student. When the gap between teacher and student is very wide, transfer can degrade, because the student has too little capacity to absorb everything that a much larger model expresses. Research on this capacity gap has found that the strongest available teacher is sometimes a poor choice. A set of methods exists to bridge wide gaps by adding a middle step, where the teacher trains a mid-sized model and that model trains the small student, so each handoff spans a smaller distance.

Architecture can outweigh size: The design of the base model can matter more than its parameter count. In one study, a 32-billion-parameter student outperformed a 70-billion-parameter student on the same task, because the smaller one was built on a stronger base architecture. Size alone is a weak predictor of how well distillation will go.

The teacher can pass on more than the task: In a 2025 study later published in Nature, a teacher model with a particular trait, a tendency to favor owls, was used to generate training data made up only of number sequences. A student trained on those numbers picked up the same preference for owls, even after the data was filtered to remove any visible trace of the trait. The same effect appeared with more serious behaviors, and it occurred only when the teacher and student shared the same base model. The takeaway is that distillation can carry across more than the task being taught, and that filtering the visible data is sometimes too coarse to stop it.

These limits set the boundaries, and within them, the method keeps advancing. The next section covers where it is heading.

The newest direction in distillation reduces the manual effort by automating the whole process.

In this setup, the large model runs the full loop on its own. It generates training data, fine-tunes the student, evaluates the student against a held-out set of examples it also generates, and repeats the cycle, adjusting what it produces until the student stops improving. The human role shrinks to defining the task and the success criteria at the start, with a final check on real data at the end.

Recent work in 2026 applied this to a detection task and found that it worked well, with one finding worth keeping in mind.

The choice of teacher model had a large effect on the outcome. Different teachers, given the same loop and the same student, produced students of noticeably different quality. So automation removes manual effort while making the initial choice of teacher more consequential, since that choice now drives an entire self-running process rather than a single training pass.

The same loop also points toward less hand-built pipeline work over time, as more of the data generation and evaluation moves to the model itself. For a team, the appeal is building a small, task-specific model without assembling a large hand-labeled dataset first, since the teacher supplies both the training examples and the data used to score them.