Does LLM Pre-Training Typically Occur at the Edge of Stability?
Quadratic approximations provide a common lens for studying neural network optimization, but recent evidence challenges their predictive validity. In full-batch gradient descent with learning rate (LR) $η$, Cohen et al. (2021) observed the Edge of Stability (EoS), where the largest Hessian eigenvalue concentrates near $2/η$, seemingly at odds with classical stability conditions predicted by quadratic approximations. In this work, we revisit the fidelity of quadratic approximations as a model of neural network training dynamics, with a particular focus on their failure modes in LLM training. We first highlight a distinct failure mode of the quadratic approximation, where persistent negativ...
| Authors | Yuhang Cai, Haodong Wen, Haofeng Huang et al. |
|---|---|
| Primary category | cs.LG |
| Published | 27 October 2026 |
| Last updated at source | 27 October 2026 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 28 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.