The Disharmony between BN and ReLU Causes Gradient Explosion, but is Offset by the Correlation between Activations

Inyoung Paik, Jaesik Choi

Deep neural networks, which employ batch normalization and ReLU-like activation functions, suffer from instability in the early stages of training due to the high gradient induced…