Superposition unifies power-law training dynamics

Zixin Jessie Chen, Hao Chen, Yizhou Liu +1

We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training…