Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
Transformer attention is typically implemented using softmax normalization, which enforces attention weights with unit sum normalization. While effective in many settings, this constraint can limit flexibility in controlling attention magnitudes and may contribute to overly concentrated or unstable attention patterns during training. Prior work has explored modifications such as attention sinks or gating mechanisms, but these approaches provide only limited or indirect control over attention reweighting. We propose Affine-Scaled Attention, a simple extension to standard attention that introduces input-dependent scaling and a corresponding bias term applied to softmax-normalized attention ...
| Authors | Jeongin Bae, Baeseong Park, Gunho Park et al. |
|---|---|
| Primary category | cs.CL |
| Published | 26 February 2026 |
| Last updated at source | 26 February 2026 |
Popularity, last 2 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- KitsuVp/NeoLLMModel · cited by
Models and datasets referencing this paper
- KitsuVp/NeoLLMModel
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 19 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.