What do Language Models Learn and When? The Implicit Curriculum Hypothesis
Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model ...
| Authors | Emmy Liu, Kaiser Sun, Millicent Li et al. |
|---|---|
| Primary category | cs.CL |
| Published | 9 April 2026 |
| Last updated at source | 9 April 2026 |
Connected in the knowledge graph
- elemental-tasks/model-trajectoriesDataset · cited by
- elemental-tasks/paper-artifactsDataset · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.