Multiscale Policy Optimization for Long-Horizon LLM Agents
Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.
| Authors | Tao Ren, Hui Yang, Jia Deng et al. |
|---|---|
| Primary category | cs.AI |
| Published | 26 October 2026 |
| Last updated at source | 2 November 2026 |
Popularity, last 14 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 9 November 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.