ShakeDrop Regularization for Deep Residual Learning

Abstract

Overfitting is a crucial problem in deep neural networks, even in the latest network architectures. In this paper, to relieve the overfitting effect of ResNet and its improvements (i.e., Wide ResNet, PyramidNet, and ResNeXt), we propose a new regularization method called ShakeDrop regularization. ShakeDrop is inspired by Shake-Shake, which is an effective regularization method, but can be applied to ResNeXt only. ShakeDrop is more effective than Shake-Shake and can be applied not only to ResNeXt but also ResNet, Wide ResNet, and PyramidNet. An important key is to achieve stability of training. Because effective regularization often causes unstable training, we introduce a training stabilizer, which is an unusual use of an existing regularizer. Through experiments under various conditions, we demonstrate the conditions under which ShakeDrop works well.

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Explaining and Harnessing Adversarial Examples
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Shake-Shake regularization
• Dataset Augmentation in Feature Space
• Deep Pyramidal Residual Networks with Separated Stochastic Depth

Cited by in corpus (54)

• Unsupervised Data Augmentation for Consistency Training
• AutoAugment: Learning Augmentation Policies from Data
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Born Again Neural Networks
• Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks
• Delving Deep into Label Smoothing
• Fixup Initialization: Residual Learning Without Normalization
• NSML: Meet the MLaaS platform with a real-world case study
• FMix: Enhancing Mixed Sample Data Augmentation
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• DADA: Differentiable Automatic Data Augmentation
• An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods
• DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Block-wise Scrambled Image Recognition Using Adaptation Network
• On Feature Normalization and Data Augmentation
• GradAug: A New Regularization Method for Deep Neural Networks
• Sample Mixed-Based Data Augmentation for Domestic Audio Tagging
• Adaptive Regularization of Labels
• ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
• Self-Knowledge Distillation with Progressive Refinement of Targets
• One-Shot Neural Architecture Search via Compressive Sensing
• KeepAugment: A Simple Information-Preserving Data Augmentation Approach
• On the Generalization Effects of Linear Transformations in Data Augmentation
• Learning data augmentation policies using augmented random search
• ContextMix: A context-aware data augmentation method for industrial visual inspection systems
• Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples
• Regularizing Neural Networks via Adversarial Model Perturbation
• Rethinking Recurrent Neural Networks and Other Improvements for Image Classification
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• Parallel Grid Pooling for Data Augmentation
• Convolutional Neural Networks with Dynamic Regularization
• Drop-Activation: Implicit Parameter Reduction and Harmonic Regularization
• Switching Transferable Gradient Directions for Query-Efficient Black-Box Adversarial Attacks
• DAAS: Differentiable Architecture and Augmentation Policy Search
• Learning Temporally Invariant and Localizable Features via Data Augmentation for Video Recognition
• Hypernetwork-Based Augmentation
• Deep Neural Network Ensembles
• Normalization Before Shaking Toward Learning Symmetrically Distributed Representation Without Margin in Speech Emotion Recognition
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Stochastic Bottleneck: Rateless Auto-Encoder for Flexible Dimensionality Reduction
• Out-distribution aware Self-training in an Open World Setting
• MetaAugment: Sample-Aware Data Augmentation Policy Learning
• Simulating Unknown Target Models for Query-Efficient Black-box Attacks
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• Parabolic Approximation Line Search for DNNs
• Elastic Neural Networks for Classification
• Effect of Various Regularizers on Model Complexities of Neural Networks in Presence of Input Noise
• Optimizing Data Augmentation Policy Through Random Unidimensional Search
• Spatially Attentive Output Layer for Image Classification