Probing the Limits of Compressive Memory: A Study of Infini-Attention in Small-Scale Pretraining

Ruizhe Huang, Kexuan Zhang, Yihao Fang +1

This study investigates small-scale pretraining for Small Language Models (SLMs) to enable efficient use of limited data and compute, improve accessibility in low-resource settings…