Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression

Mingsong Yan, Dongyang Li, Charles Kulick +1

In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show, both theoretically and empirically, that a standard softmax-attention transformer c…