Proteus: Incremental Memory Activation for Long-Context Sequence Modeling

Reza Bayat, Ali Behrouz, Vahab Mirrokni +1

The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact sta…