On the Interpolation Error of Nonlinear Attention versus Linear Regression

Zhenyu Liao, Jiaqing Liu, TianQi Hou +2

Attention has become the core building block of modern machine learning (ML) by efficiently capturing the long-range dependencies among input tokens. Its inherently parallelizable…