TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents
On-policy distillation (OPD) has shown strong potential for transferring reasoning ability from frontier or domain-specific models to smaller students. While effective on static single-turn tasks, its behavior in multi-turn agent settings remains underexplored. In this work, we identify a key limitation of vanilla OPD in such settings, which we term Trajectory-Level KL Instability. Specifically, we observe that KL divergence increases together with a drop in success rate, and even after convergence, the KL remains high, leading to unstable training. This instability arises from inter-turn error compounding: as errors accumulate, the student is driven beyond the teacher's effective support...
| Authors | Jiaqi Wang, Wenhao Zhang, Weijie Shi et al. |
|---|---|
| Primary category | cs.LG |
| Published | 27 April 2026 |
| Last updated at source | 29 April 2026 |
Connected in the knowledge graph
- SeanWang0027/Multi-TurnDataset · cited by
Models and datasets referencing this paper
- SeanWang0027/Multi-TurnDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.