Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers

Thiziri Nait Saada, Alireza Naderi, Jared Tanner

Attention layers are the core component of transformers, the current state-of-the-art neural network architecture. Alternatives to softmax-based attention are being explored due to…