Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models. A practical way to obtain such models is to convert a pretrained Transformer instead of pretraining a new architecture from scratch, but this conversion is still brittle. Simply copying the teacher attention projections into a Gated DeltaNet (GDN) student does not specify the new recurrent decay, write, and output-gating dynamics. As a result, the converted model often starts in a poor dynamical regime and must spend many distillation tokens repairing initialization r...
| Authors | Zhongzhu Zhou, Qingyang Wu, Junxiong Wang et al. |
|---|---|
| Primary category | cs.LG |
| Published | 15 June 2026 |
| Last updated at source | 15 June 2026 |
Popularity, last 3 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- modulon-labs/K2-Horizon-7B-GDN-Hybrid-ExpModel · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 19 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.