Learning the identity: a case study of how SGD selects among functional decompositions

Andy Arditi, Weian Xie, David Bau +1

One might think that learning the identity function with a deep linear residual network is trivial - the path along residual connections already implements the identity, and so the…