ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection

Huy Tran, Max Milkert, David Hyde

Doubly-stochastic attention has emerged as a transport-based alternative to row-softmax attention, with recent Transformer variants using it to reduce attention sinks and rank coll…