Gradient Centralization: A New Optimization Technique for Deep Neural Networks

Abstract

Optimization techniques are of great importance to effectively and efficiently train a deep neural network (DNN). It has been shown that using the first and second order statistics (e.g., mean and variance) to perform Z-score standardization on network activations or weight vectors, such as batch normalization (BN) and weight standardization (WS), can improve the training performance. Different from these existing methods that mostly operate on activations or weights, we present a new optimization technique, namely gradient centralization (GC), which operates directly on gradients by centralizing the gradient vectors to have zero mean. GC can be viewed as a projected gradient descent method with a constrained loss function. We show that GC can regularize both the weight space and output feature space so that it can boost the generalization performance of DNNs. Moreover, GC improves the Lipschitzness of the loss function and its gradient so that the training process becomes more efficient and stable. GC is very simple to implement and can be easily embedded into existing gradient based DNN optimizers with only one line of code. It can also be directly used to fine-tune the pre-trained DNNs. Our experiments on various applications, including general image classification, fine-grained image classification, detection and segmentation, demonstrate that GC can consistently improve the performance of DNN learning. The code of GC can be found at https://github.com/Yonghongwei/Gradient-Centralization.

20 pages, 7 figures, conference

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Deep Learning with Differential Privacy
• On the difficulty of training Recurrent Neural Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Fine-Grained Visual Classification of Aircraft
• Understanding deep learning requires rethinking generalization
• YouTube-8M: A Large-Scale Video Classification Benchmark
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Three Mechanisms of Weight Decay Regularization
• A Projected Gradient Descent Method for CRF Inference allowing End-To-End Training of Arbitrary Pairwise Potentials

Cited by in corpus (6)

• 1st Place Solution for ICDAR 2021 Competition on Mathematical Formula Detection
• A Novel lightweight Convolutional Neural Network, ExquisiteNetV2
• Urban Sound Classification : striving towards a fair comparison
• UMLE: Unsupervised Multi-discriminator Network for Low Light Enhancement
• Knowledge Generation -- Variational Bayes on Knowledge Graphs
• Delta Sampling R-BERT for limited data and low-light action recognition