Continual Learning Mechanisms Compose for Long-Horizon Memorization
Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memorization, a setting in which a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier training examples or receiving task identifiers at inference. Sequential updates cause catastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon. We hypothesize that mechanisms addressing complementary sources of forgetting will be more effective when composed. We organize these compositions along two design dimensions....
| Authors | Zheyuan Zhang, Alvin Zhang, Daniel Khashabi et al. |
|---|---|
| Primary category | cs.LG |
| Published | 7 September 2026 |
| Last updated at source | 7 September 2026 |
Connected in the knowledge graph
- cozzyde/long-horizon-memorizationDataset · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.