GradAug: A New Regularization Method for Deep Neural Networks
arXiv:2006.07989
Abstract
We propose a new regularization method to alleviate over-fitting in deep neural networks. The key idea is utilizing randomly transformed training samples to regularize a set of sub-networks, which are originated by sampling the width of the original network, in the training process. As such, the proposed method introduces self-guided disturbances to the raw gradients of the network and therefore is termed as Gradient Augmentation (GradAug). We demonstrate that GradAug can help the network learn well-generalized and more diverse representations. Moreover, it is easy to implement and can be applied to various structures and applications. GradAug improves ResNet-50 to 78.79% on ImageNet classification, which is a new state-of-the-art accuracy. By combining with CutMix, it further boosts the performance to 79.67%, which outperforms an ensemble of advanced training tricks. The generalization ability is evaluated on COCO object detection and instance segmentation where GradAug significantly surpasses other state-of-the-art methods. GradAug is also robust to image distortions and FGSM adversarial attacks and is highly effective in low data regimes. Code is available at https://github.com/taoyang1122/GradAug
arXiv commentAccepted to NeurIPS 2020
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Improved Regularization of Convolutional Neural Networks with Cutout
- Wide Residual Networks
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- FractalNet: Ultra-Deep Neural Networks without Residuals
- DropBlock: A regularization method for convolutional networks
- Shake-Shake regularization
- Adding Gradient Noise Improves Learning for Very Deep Networks
- ShakeDrop Regularization for Deep Residual Learning
- Cogradient Descent for Bilinear Optimization
Cited by in corpus (4)
- Regularized Deep Networks in Intelligent Transportation Systems: A Taxonomy and a Case Study
- Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks
- RoPGen: Towards Robust Code Authorship Attribution via Automatic Coding Style Transformation
- Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning