Lag-Relative Sparse Attention In Long Context Training

Manlai Liang, Wanyi Huang, Mandi Liu +2

Large Language Models (LLMs) have made significant strides in natural language processing and generation, yet their ability to handle long-context input remains constrained by the…