90 citations · 108 across the 5 of their papers we have counts for
9 papers
Understanding the Failure of Batch Normalization for Transformers in NLP
Jiaxi Wang, Ji Wu, Lei Huang
Batch Normalization (BN) is a core and prevalent technique in accelerating the training of deep neural networks and improving the generalization on Computer Vision (CV) tasks. Howe…
Bi-level Doubly Variational Learning for Energy-based Latent Variable Models
Ge Kan, Jinhu Lü, Tian Wang +5
Energy-based latent variable models (EBLVMs) are more expressive than conventional energy-based models. However, its potential on visual tasks are limited by its training process b…
Delving into the Estimation Shift of Batch Normalization in a Network
Lei Huang, Yi Zhou, Tian Wang +2
Batch normalization (BN) is a milestone technique in deep learning. It normalizes the activation using mini-batch statistics during training but the estimated population statistics…
Controllable Orthogonalization in Training DNNs
Lei Huang, Li Liu, Fan Zhu +4
Orthogonality is widely used for training deep neural networks (DNNs) due to its ability to maintain all singular values of the Jacobian close to 1 and reduce redundancy in represe…
An Investigation into the Stochasticity of Batch Whitening
Lei Huang, Lei Zhao, Yi Zhou +3
Batch Normalization (BN) is extensively employed in various network architectures by performing standardization within mini-batches. A full understanding of the process has been a…
Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
Lei Huang, Jie Qin, Li Liu +2
Conditioning analysis uncovers the landscape of an optimization objective by exploring the spectrum of its curvature matrix. This has been well explored theoretically for linear mo…