Backward Gradient Normalization in Deep Neural Networks

Alejandro Cabana, Luis F. Lago-Fernández

We introduce a new technique for gradient normalization during neural network training. The gradients are rescaled during the backward pass using normalization layers introduced at…