The Loss Does Not See the Basis, but Adam Does

Devender Singh

Gradient descent on a factored model $W=U{V}^{\top }$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the dif…