Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training
In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated to the model and reward density where it is most informative. We identify a reward-density principle: sparse sequence-level reward is most useful on models that can explore and discover better behavior, while dense token-level teacher supervision is better suited for compressing that behavior into a smaller deployment model; accordingly, scarce labeled data should be used upstream on the strongest available teacher, then transferred downstream as dense supervision. We evaluate this rule through a four-stage workflow:teacher RL, forward-KL warmup, on-policy distillation, o...
| Authors | Yuanda Xu, Hejian Sang, Zhengze Zhou et al. |
|---|---|
| Primary category | cs.LG |
| Published | 12 May 2026 |
| Last updated at source | 12 October 2026 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.