Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization

Taesun Yeom, Taehyeok Ha, Jaeho Lee

Feature learning strength (FLS), i.e., the inverse of the effective output scaling of a model, plays a critical role in shaping the optimization dynamics of neural nets. While its…