Batch Normalization Is Blind to the First and Second Derivatives of the Loss

Zhanpeng Zhou, Wen Shen, Huixin Chen +2

In this paper, we prove the effects of the BN operation on the back-propagation of the first and second derivatives of the loss. When we do the Taylor series expansion of the loss…