Model Distillation

Model distillation is a compression technique in which a small "student" model is trained to reproduce the behaviour of a larger "teacher" model. Instead of learning only from ground-truth labels, the student learns from the teacher's outputs, such as its full probability distributions or generated answers.

Why it matters

Large models are expensive to run and slow to respond. A distilled model can run on cheaper hardware, fit on a phone or serve many more requests per second, while retaining a useful share of the teacher's ability on a defined task. It is most effective when the target task is narrow. Terms of service of some providers restrict using their model outputs to train competing models, so check licensing.

Example

1. Run a large model over 100,000 support tickets to produce labels
2. Train a small classifier on tickets + those labels
3. Deploy the small model: faster and cheaper per request

The student rarely matches the teacher everywhere, and it inherits the teacher's mistakes.

Common mistakes

• Expecting a small model to match the teacher on tasks outside the distillation data.
• Distilling from low-quality or unchecked teacher outputs.
• Confusing it with quantization, which shrinks the same model's numbers rather than training a new one.

Related

• Quantization
• Fine-tuning
• Inference
• Model parameters

यह पेज सामान्य जानकारी है, जिसे हमारी संपादकीय टीम ने लिखा है और दस्तावेज़ीकरण बदलने पर अपडेट किया जाता है। सर्वर और प्रदाता अलग-अलग होते हैं, इसलिए बदलाव पहले सुरक्षित तरीके से जाँचें। देखें: संपादकीय नीति और अस्वीकरण.