DropCluster: A structured dropout for convolutional networks

Abstract

Dropout as a common regularizer to prevent overfitting in deep neural networks has been less effective in convolutional layers than in fully connected layers. This is because Dropout drops features randomly, without considering local structure. When features are spatially correlated, as in the case of convolutional layers, information from the dropped features can still propagate to subsequent layers via neighboring features. To address this problem, structured forms of Dropout have been proposed. A drawback of these methods is that they do not adapt to the data. In this work, we leverage the structure in the outputs of convolutional layers and introduce a novel structured regularization method named DropCluster. Our approach clusters features in convolutional layers, and drops the resulting clusters randomly during training iterations. Experiments on CIFAR-10/100, SVHN, and APPA-REAL datasets demonstrate that our approach is effective and controls overfitting better than other approaches.

11 pages, 10 figures, under review

References in corpus (12)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• The NumPy array: a structure for efficient numerical computation
• DropBlock: A regularization method for convolutional networks
• Deep Anomaly Detection with Outlier Exposure
• Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Early Methods for Detecting Adversarial Images
• Open Category Detection with PAC Guarantees
• Confidence Calibration for Convolutional Neural Networks Using Structured Dropout
• Recursive nearest agglomeration (ReNA): fast clustering for approximation of structured signals
• Feature Grouping as a Stochastic Regularizer for High-Dimensional Structured Data

Cited by in corpus (4)

• HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• Partial Graph Reasoning for Neural Network Regularization
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks