Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models. However, existing OPD distillation methods require teacher and student models to share the same tokenizer, restricting the applicability of OPD within the model series. Current mainstream practice typically employs Supervised Fine-Tuning (SFT) on teacher-generated responses for cross-tokenizer distillation, which fails to capture the rich knowledge embedded in the teacher's probability distribution. In this work, we enable the standard on-policy distillation method to operate across model families, ensuring that hig...
| Authors | Yifan Niu, Han Xiao, Dongyi Liu et al. |
|---|---|
| Primary category | cs.LG |
| Published | 8 June 2026 |
| Last updated at source | 8 June 2026 |
Connected in the knowledge graph
- codemaivanngu/dpca-b200-uvlock-cu128Dataset · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.