Sinkhorn doubly stochastic attention rank decay analysis

Michela Lapenna, Rita Fioresi, Bahman Gharesifard

The self-attention mechanism is central to the success of Transformer architectures. However, standard row-stochastic attention has been shown to suffer from significant signal deg…