Correlation-Aware Select and Merge Attention for Efficient Fine-Tuning and Context Length Extension

Ning Wang, Zekun Li, Tongxin Bai +1

Modeling long sequences is crucial for various large-scale models; however, extending existing architectures to handle longer sequences presents significant technical and resource…