RotationOut as a Regularization Method for Neural Network
arXiv:1911.07427
Abstract
In this paper, we propose a novel regularization method, RotationOut, for neural networks. Different from Dropout that handles each neuron/channel independently, RotationOut regards its input layer as an entire vector and introduces regularization by randomly rotating the vector. RotationOut can also be used in convolutional layers and recurrent layers with small modifications. We further use a noise analysis method to interpret the difference between RotationOut and Dropout in co-adaptation reduction. Using this method, we also show how to use RotationOut/Dropout together with Batch Normalization. Extensive experiments in vision and language tasks are conducted to show the effectiveness of the proposed method. Codes are available at \url{https://github.com/RotationOut/RotationOut}.
arXiv comment20 pages, 8 figures
References in corpus (11)
- Improving neural networks by preventing co-adaptation of feature detectors
- Improved Regularization of Convolutional Neural Networks with Cutout
- Recurrent Neural Network Regularization
- Wide Residual Networks
- Focal Loss for Dense Object Detection
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- FractalNet: Ultra-Deep Neural Networks without Residuals
- DropBlock: A regularization method for convolutional networks
- Regularizing and Optimizing LSTM Language Models
- Shake-Shake regularization
- Survey of Dropout Methods for Deep Neural Networks