Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer

Yihe Dong, Lorenzo Noci, Mikhail Khodak +1

The transformer architecture is central to the success of modern Large Language Models (LLMs), in part due to its surprising ability to perform a wide range of tasks - including ma…