Linear RNN Scaling Laws: When Longer Sequences Beat More Sequences

Ziyan Chen, Zhongzhu Zhou, Peilin Liu +2

Empirical scaling laws for autoregressive language models relate prediction loss to model size, data size, and optimization compute, but their theoretical origin is still poorly un…