LongNet: Scaling Transformers to 1,000,000,000 Tokens

Jiayu Ding, Shuming Ma, Li Dong +5

Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressiv…