Drop-Activation: Implicit Parameter Reduction and Harmonic Regularization
arXiv:1811.05850
Abstract
Overfitting frequently occurs in deep learning. In this paper, we propose a novel regularization method called Drop-Activation to reduce overfitting and improve generalization. The key idea is to drop nonlinear activation functions by setting them to be identity functions randomly during training time. During testing, we use a deterministic network with a new activation function to encode the average effect of dropping activations randomly. Our theoretical analyses support the regularization effect of Drop-Activation as implicit parameter reduction and verify its capability to be used together with Batch Normalization (Ioffe and Szegedy 2015). The experimental results on CIFAR-10, CIFAR-100, SVHN, EMNIST, and ImageNet show that Drop-Activation generally improves the performance of popular neural network architectures for the image classification task. Furthermore, as a regularizer Drop-Activation can be used in harmony with standard training and regularization techniques such as Batch Normalization and Auto Augment (Cubuk et al. 2019). The code is available at \url{https://github.com/LeungSamWai/Drop-Activation}.
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- mixup: Beyond Empirical Risk Minimization
- Improved Regularization of Convolutional Neural Networks with Cutout
- Empirical Evaluation of Rectified Activations in Convolutional Network
- Wide Residual Networks
- Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
- EMNIST: an extension of MNIST to handwritten letters
- Shake-Shake regularization
- Dropout Training as Adaptive Regularization
- ShakeDrop Regularization for Deep Residual Learning
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Swapout: Learning an ensemble of deep architectures