Broken scale symmetries in undercomplete linear autoencoders

Farhad Pashakhanloo, Jacob A. Zavatone-Veth

Neural network loss landscapes have many symmetries, which are preserved by gradient flow but broken by finite-stepsize stochastic gradient descent (SGD). A canonical example of su…