The lazy (NTK) and rich ($\mu$P) regimes: a gentle tutorial

Dhruva Karkada

A central theme of the modern machine learning paradigm is that larger neural networks achieve better performance on a variety of metrics. Theoretical analyses of these overparamet…