Low-rank bias, weight decay, and model merging in neural networks

Ilja Kuzborskij, Yasin Abbasi Yadkori

We explore the low-rank structure of the weight matrices in neural networks at the stationary points (limiting solutions of optimization algorithms) with $L2$ regularization (also…