The Optimization Landscape of SGD Across the Feature Learning Strength

Alexander Atanasov, Alexandru Meterez, James B. Simon +1

We consider neural networks (NNs) where the final layer is down-scaled by a fixed hyperparameter $\gamma$. Recent work has identified $\gamma$ as controlling the strength of feature learning…