Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show, both theoretically and empirically, that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass. Under bounded-data assumptions, we construct a single-head transformer whose forward pass approximately implements \textit{preconditioned Richardson iteration} on the associated kernel system. The construction uses $O(\log(1/ε))$ blocks and MLP width $O(\sqrt{N/ε})$ to achieve $ε$-accurate prediction for prompts of length $N$. Our construction reveals a functional decomposition within the transformer architecture: softmax attention produces a row-normaliz...
| Authors | Mingsong Yan, Dongyang Li, Charles Kulick et al. |
|---|---|
| Primary category | cs.LG |
| Published | 8 May 2026 |
| Last updated at source | 25 September 2026 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.