Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments
An agent's performance stagnating at a suboptimal level is a common problem in deep on-policy RL. Focusing on PPO, we show that plateaus in certain regimes arise not because of known exploration, capacity, or optimisation challenges, but because sample-based estimates of the loss eventually become poor proxies for the true objective over the course of training. Looking deeper, PPO alternates between sampling rollouts from several parallel environments online using the current policy (which we call the "outer loop") and performing repeated minibatch SGD steps against this offline dataset (the "inner loop"). In our work, we abstract away the inner loop, and conceptually model the outer loop...
| Authors | Michael Beukman, Khimya Khetarpal, Zeyu Zheng et al. |
|---|---|
| Primary category | cs.LG |
| Published | 6 March 2026 |
| Last updated at source | 18 July 2026 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.