FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

Zekun Shi, Alastair James Arthur Price

Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures…