The Model Shape Behind Scaling Laws: A Gradient Superposition Perspective
Neural scaling laws guide resource allocation, but how should a fixed parameter budget be divided between depth and width? Motivated by inverse-width representation-error scaling under strong superposition, we study this allocation through the network's end-to-end response. The Neural Feature Ansatz links weight and gradient geometry, motivating the Frobenius norm $F$ of the input average gradient outer product (AGOP) as an observable of response strength and overlap. By reproducing Anthropic's toy-model double-descent experiment, we find that $F$ and test reconstruction loss follow an approximately affine relationship across the full sample-size sweep and share an interpolation peak. Thr...
| Authors | Wenjie Sun, Jinning Yang, Shuai Zhang et al. |
|---|---|
| Primary category | cs.LG |
| Published | 27 May 2026 |
| Last updated at source | 3 October 2026 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.