Ordinary Least Squares is a Special Case of Transformer

Xiaojun Tan, Yuchen Zhao

The statistical essence of the Transformer architecture has long remained elusive: Is it a universal approximator, or a neural network version of known computational algorithms? Th…