Ordinary Least Squares as an Attention Mechanism

Philippe Goulet Coulombe

I show that ordinary least squares (OLS) predictions can be rewritten as the output of a restricted attention module, akin to those forming the backbone of large language models. T…