Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD). This recovery step largely decides the final quality, yet it is expensive. We present a practitioner's study of how to make distillation training efficient, organised around two systems contributions. First, we show that offline KD (caching the teacher's top-$K$ logits once and training the student against the cache) matches online distillation at near-identical training loss while removing the teacher from memory, running about 29\% faster per iteration,...
| Authors | Bakbergen Ryskulov, Iker García-Ferrero, David Montero et al. |
|---|---|
| Primary category | cs.CL |
| Published | 4 August 2026 |
| Last updated at source | 4 August 2026 |
Popularity, last 2 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- MultiverseComputingCAI/LittleLambModel · cited by
- MultiverseComputingCAI/Hypernova-60B-2605Model · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.