Shake-Shake regularization

Abstract

The method introduced in this paper aims at helping deep learning practitioners faced with an overfit problem. The idea is to replace, in a multi-branch network, the standard summation of parallel branches with a stochastic affine combination. Applied to 3-branch residual networks, shake-shake regularization improves on the best single shot published results on CIFAR-10 and CIFAR-100 by reaching test errors of 2.86% and 15.85%. Experiments on architectures without skip connections or Batch Normalization show encouraging results and open the door to a large set of applications. Code is available at https://github.com/xgastaldi/shake-shake

References in corpus (8)

• Densely Connected Convolutional Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Aggregated Residual Transformations for Deep Neural Networks
• Deep Networks with Stochastic Depth
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Convergent Learning: Do different neural networks learn the same representations?
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks

Cited by in corpus (140)

• Improved Regularization of Convolutional Neural Networks with Cutout
• Neural Architecture Search: A Survey
• Unsupervised Data Augmentation for Consistency Training
• AutoAugment: Learning Augmentation Policies from Data
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• Born Again Neural Networks
• Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks
• Robustness May Be at Odds with Accuracy
• GridMask Data Augmentation
• Averaging Weights Leads to Wider Optima and Better Generalization
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• A Survey on Neural Architecture Search
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• A Review on Deep Learning in Medical Image Reconstruction
• ShakeDrop Regularization for Deep Residual Learning
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• Path-Level Network Transformation for Efficient Architecture Search
• Weighted Transformer Network for Machine Translation
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Selective Kernel Networks
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
• Best Practices for Scientific Research on Neural Architecture Search
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
• MaskConnect: Connectivity Learning by Gradient Descent
• SELF: Learning to Filter Noisy Labels with Self-Ensembling
• Multi-Sample Dropout for Accelerated Training and Better Generalization
• Adaptive Boosting for Domain Adaptation: Towards Robust Predictions in Scene Segmentation
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• Confidence Calibration for Convolutional Neural Networks Using Structured Dropout
• ResizeMix: Mixing Data with Preserved Object Information and True Labels
• UniformAugment: A Search-free Probabilistic Data Augmentation Approach
• An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
• DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Improving Auto-Augment via Augmentation-Wise Weight Sharing
• Optimized Generic Feature Learning for Few-shot Classification across Domains
• Connectivity Learning in Multi-Branch Networks
• GradAug: A New Regularization Method for Deep Neural Networks
• Regularizing Deep Networks with Semantic Data Augmentation
• Efficient Certified Defenses Against Patch Attacks on Image Classifiers
• Sample Mixed-Based Data Augmentation for Domestic Audio Tagging
• Quantitative Phase Imaging and Artificial Intelligence: A Review
• CIFAR10 to Compare Visual Recognition Performance between Deep Neural Networks and Humans
• Capsule Networks Need an Improved Routing Algorithm
• Parle: parallelizing stochastic gradient descent
• Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Need
• Robust Learning Under Label Noise With Iterative Noise-Filtering
• Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
• Semi-Supervised Learning by Label Gradient Alignment
• EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations
• EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks
• MEAL: Multi-Model Ensemble via Adversarial Learning
• On Training Implicit Models
• KeepAugment: A Simple Information-Preserving Data Augmentation Approach
• Group Ensemble: Learning an Ensemble of ConvNets in a single ConvNet
• On the Generalization Effects of Linear Transformations in Data Augmentation
• Deep Metric Transfer for Label Propagation with Limited Annotated Data
• Learning data augmentation policies using augmented random search
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• ATHENA: A Framework based on Diverse Weak Defenses for Building Adversarial Defense
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning
• Depthwise Multiception Convolution for Reducing Network Parameters without Sacrificing Accuracy
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
• Hierarchical Neural Architecture Search via Operator Clustering
• Triple Generative Adversarial Networks
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• CacheNet: A Model Caching Framework for Deep Learning Inference on the Edge
• A Novel Framework for Neural Architecture Search in the Hill Climbing Domain
• Between-class Learning for Image Classification
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality
• On the Effectiveness of Regularization Against Membership Inference Attacks
• Every Node Counts: Self-Ensembling Graph Convolutional Networks for Semi-Supervised Learning
• Drop-Activation: Implicit Parameter Reduction and Harmonic Regularization
• AdaSGD: Bridging the gap between SGD and Adam
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Convolutional Neural Networks with Dynamic Regularization
• Repetitive Reprediction Deep Decipher for Semi-Supervised Learning
• You Look Twice: GaterNet for Dynamic Filter Selection in CNNs
• On the Blindspots of Convolutional Networks
• Intra-Ensemble in Neural Networks
• Learning Optimal Data Augmentation Policies via Bayesian Optimization for Image Classification Tasks
• OnlineAugment: Online Data Augmentation with Less Domain Knowledge
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
• Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers
• Bag of Tricks for Neural Architecture Search
• One-bit Supervision for Image Classification
• Image recognition from raw labels collected without annotators
• Learning Temporally Invariant and Localizable Features via Data Augmentation for Video Recognition
• Effective Evaluation of Deep Active Learning on Image Classification Tasks
• Gradually Updated Neural Networks for Large-Scale Image Recognition
• Deep Networks with Fast Retraining
• Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement
• Direct Differentiable Augmentation Search
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond
• Normalization Before Shaking Toward Learning Symmetrically Distributed Representation Without Margin in Speech Emotion Recognition
• Fine-tuning Handwriting Recognition systems with Temporal Dropout
• Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning
• Genetic Network Architecture Search
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Hypernetwork-Based Augmentation
• On the Similarity of Deep Learning Representations Across Didactic and Adversarial Examples
• Local Patch AutoAugment with Multi-Agent Collaboration
• EENA: Efficient Evolution of Neural Architecture
• Variations on the Chebyshev-Lagrange Activation Function
• BasisNet: Two-stage Model Synthesis for Efficient Inference
• Meta Gradient Adversarial Attack
• Multi-level Feature Fusion-based CNN for Local Climate Zone Classification from Sentinel-2 Images: Benchmark Results on the So2Sat LCZ42 Dataset
• Shaking Acoustic Spectral Sub-bands Can Better Regularize Learning in Affective Computing
• LA3: Efficient Label-Aware AutoAugment
• MetaAugment: Sample-Aware Data Augmentation Policy Learning
• Effect of Various Regularizers on Model Complexities of Neural Networks in Presence of Input Noise
• SwGridNet: A Deep Convolutional Neural Network based on Grid Topology for Image Classification
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor
• Semi-Supervised Learning with Meta-Gradient
• Multi-Scale Spatially-Asymmetric Recalibration for Image Classification
• Interlayer and Intralayer Scale Aggregation for Scale-invariant Crowd Counting
• Reappraising Domain Generalization in Neural Networks
• Parabolic Approximation Line Search for DNNs
• Adaptive Perturbation for Adversarial Attack
• Improve SGD Training via Aligning Mini-batches
• RotationOut as a Regularization Method for Neural Network
• One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
• Selective Output Smoothing Regularization: Regularize Neural Networks by Softening Output Distributions
• Sequenced-Replacement Sampling for Deep Learning
• Unsupervised Representation Learning with Laplacian Pyramid Auto-encoders
• Hierarchical Auxiliary Learning
• Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization
• FROST: Faster and more Robust One-shot Semi-supervised Training
• A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning