LocalDrop: A Hybrid Regularization for Deep Neural Networks

Abstract

In neural networks, developing regularization algorithms to settle overfitting is one of the major study areas. We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs), including drop rates and weight matrices, has been developed based on the proposed upper bound of the local Rademacher complexity by the strict mathematical deduction. The analyses of dropout in FCNs and DropBlock in CNNs with keep rate matrices in different layers are also included in the complexity analyses. With the new regularization function, we establish a two-stage procedure to obtain the optimal keep rate matrix and weight matrix to realize the whole training model. Extensive experiments have been conducted to demonstrate the effectiveness of LocalDrop in different models by comparing it with several algorithms and the effects of different hyperparameters on the final performances.

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Improved Regularization of Convolutional Neural Networks with Cutout
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Wide Residual Networks
• DropBlock: A regularization method for convolutional networks
• Local Rademacher complexities
• Fractional Max-Pooling
• Variational Dropout Sparsifies Deep Neural Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Norm-Based Capacity Control in Neural Networks
• A Spatial Layout and Scale Invariant Feature Representation for Indoor Scene Classification
• Variational Bayesian dropout: pitfalls and fixes
• On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyond

Cited by in corpus (3)

• Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks
• Dynamic Evolution of Complex Networks: A Reinforcement Learning Approach Applying Evolutionary Games to Community Structure
• Adaptive Low-Rank Regularization with Damping Sequences to Restrict Lazy Weights in Deep Networks