MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization

Wen Fei, Wenrui Dai, Chenglin Li +2

Substantial experiments have validated the success of Batch Normalization (BN) Layer in benefiting convergence and generalization. However, BN requires extra memory and float-point…