Unshackling Context Length: An Efficient Selective Attention Approach through Query-Key Compression

Haoyu Wang, Tong Teng, Tianyu Guo +4

Handling long-context sequences efficiently remains a significant challenge in large language models (LLMs). Existing methods for token selection in sequence extrapolation either e…