Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Oliver Sieberling, Bharat Runwal, David Jin +3

Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial fa…