Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization

Brendan Ruff, Taylor Beck, Joscha Bach

Deep convolutional neural networks are known to be unstable during training at high learning rate unless normalization techniques are employed. Normalizing weights or activations a…