Sparse Sinkhorn Attention

Yi Tay, Dara Bahri, Liu Yang +2

We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend. Our method is based on differentiable sorting of internal representations. Concretel…