On the Nonlinearity of Layer Normalization

Yunhao Ni, Yuxin Guo, Junlong Jia +1

Layer normalization (LN) is a ubiquitous technique in deep learning but our theoretical understanding to it remains elusive. This paper investigates a new theoretical direction for…