Critical Data Size of Language Models from a Grokking Perspective

Xuekai Zhu, Yao Fu, Bowen Zhou +1

We explore the critical data size in language models, a threshold that marks a fundamental shift from quick memorization to slow generalization. We formalize the phase transition u…