RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers

Md Zesun Ahmed Mia, Malyaban Bal, Abhronil Sengupta

The quadratic complexity of self-attention mechanism presents a significant impediment to applying Transformer models to long sequences. This work explores computational principles…