Understanding the Failure of Batch Normalization for Transformers in NLP

Jiaxi Wang, Ji Wu, Lei Huang

Batch Normalization (BN) is a core and prevalent technique in accelerating the training of deep neural networks and improving the generalization on Computer Vision (CV) tasks. Howe…