Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks

Hadi Daneshmand, Jonas Kohler, Francis Bach +2

Randomly initialized neural networks are known to become harder to train with increasing depth, unless architectural enhancements like residual connections and batch normalization…