Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence

Vinay Rao, Jascha Sohl-Dickstein

We perform an extensive empirical study of the statistical properties of Batch Norm and other common normalizers. This includes an examination of the correlation between representa…