SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

Zian Liu, Yiwen Hu, Zican Dong +5

Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typ…