Adaptive Memory Decay for Log-Linear Attention

Yaxita Amin, Helen Zichen Li, Mengfan Zhang +1

Sequence models face a fundamental tradeoff between memory capacity and computational efficiency. Transformers achieve expressive context modeling at quadratic cost, while linear a…