An ETF view of Dropout regularization
arXiv:1810.06049
Abstract
Dropout is a popular regularization technique in deep learning. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain family of autoencoders with a linear encoder, optimizing the encoder with dropout regularization leads to an equiangular tight frame (ETF). Since this optimization is non-convex, we add another regularization that promotes such structures by minimizing the cross-correlation between filters in the network. We demonstrate its applicability in convolutional and fully connected layers in both feed-forward and recurrent networks. All these results suggest that there is indeed a relationship between dropout and ETF structure of the regularized linear operations.
arXiv commentAccepted to BMVC 2020
References in corpus (19)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Auto-Encoding Variational Bayes
- Improving neural networks by preventing co-adaptation of feature detectors
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Recurrent Neural Network Regularization
- DropBlock: A regularization method for convolutional networks
- Group Sparse Regularization for Deep Neural Networks
- Variational Autoencoder for Deep Learning of Images, Labels and Captions
- Dropout Training as Adaptive Regularization
- Robust Large Margin Deep Neural Networks
- Regularization for Deep Learning: A Taxonomy
- Reducing Overfitting in Deep Networks by Decorrelating Representations
- Norm-Based Capacity Control in Neural Networks
- Finite tight frames and some applications
- On the Implicit Bias of Dropout
- Altitude Training: Strong Bounds for Single-Layer Dropout
- Dropout as a Low-Rank Regularizer for Matrix Factorization
- Random Subsets of Structured Deterministic Frames have MANOVA Spectra
- Dropout is a special case of the stochastic delta rule: faster and more accurate deep learning