Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks

Abstract

Several image processing tasks, such as image classification and object detection, have been significantly improved using Convolutional Neural Networks (CNN). Like ResNet and EfficientNet, many architectures have achieved outstanding results in at least one dataset by the time of their creation. A critical factor in training concerns the network's regularization, which prevents the structure from overfitting. This work analyzes several regularization methods developed in the last few years, showing significant improvements for different CNN models. The works are classified into three main areas: the first one is called "data augmentation", where all the techniques focus on performing changes in the input data. The second, named "internal changes", which aims to describe procedures to modify the feature maps generated by the neural network or the kernels. The last one, called "label", concerns transforming the labels of a given input. This work presents two main differences comparing to other available surveys about regularization: (i) the first concerns the papers gathered in the manuscript, which are not older than five years, and (ii) the second distinction is about reproducibility, i.e., all works refered here have their code available in public repositories or they have been directly implemented in some framework, such as TensorFlow or Torch.

arXiv comment27 pages

References in corpus (25)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• mixup: Beyond Empirical Risk Minimization
• Generative Adversarial Networks
• In Defense of the Triplet Loss for Person Re-Identification
• Improved Regularization of Convolutional Neural Networks with Cutout
• Squeeze-and-Excitation Networks
• Person Re-identification: Past, Present and Future
• AutoAugment: Learning Augmentation Policies from Data
• Shake-Shake regularization
• A Survey on Methods and Theories of Quantized Neural Networks
• ShakeDrop Regularization for Deep Residual Learning
• Augment your batch: better training with larger batches
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• Meta Pseudo Labels
• Towards Understanding Label Smoothing
• Regularization via Structural Label Smoothing
• GradAug: A New Regularization Method for Deep Neural Networks
• Dropout as a Low-Rank Regularizer for Matrix Factorization
• LocalDrop: A Hybrid Regularization for Deep Neural Networks
• TargetDrop: A Targeted Regularization Method for Convolutional Neural Networks

Cited by in corpus (5)

• ECRECer: Enzyme Commission Number Recommendation and Benchmarking based on Multiagent Dual-core Learning
• Identifying Bias in Deep Neural Networks Using Image Transforms
• Certified Control for Train Sign Classification
• Recurrent Transformer-Based Near- and Far-Field THz Wideband Channel Estimation for UM-MIMO
• A solution to generalized learning from small training sets found in infants repeated visual experiences of individual objects