Multi-Sample Dropout for Accelerated Training and Better Generalization

Abstract

Dropout is a simple but efficient regularization technique for achieving better generalization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards a portion of the neurons to avoid overfitting. This paper presents an enhanced dropout technique, which we call multi-sample dropout, for both accelerating training and improving generalization over the original dropout. The original dropout creates a randomly selected subset (called a dropout sample) from the input in each training iteration while the multi-sample dropout creates multiple dropout samples. The loss is calculated for each sample, and then the sample losses are averaged to obtain the final loss. This technique can be easily implemented by duplicating a part of the network after the dropout layer while sharing the weights among the duplicated fully connected layers. Experimental results using image classification tasks including ImageNet, CIFAR-10, and CIFAR-100 showed that multi-sample dropout accelerates training. Moreover, the networks trained using multi-sample dropout achieved lower error rates compared to networks trained with the original dropout. The additional computation cost due to the duplicated operations is not significant for deep convolutional networks because most of the computation time is consumed in the convolution layers before the dropout layer, which are not duplicated.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SMOTE: Synthetic Minority Over-sampling Technique
• Improving neural networks by preventing co-adaptation of feature detectors
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• AutoAugment: Learning Augmentation Policies from Data
• DropBlock: A regularization method for convolutional networks
• Data Augmentation by Pairing Samples for Images Classification
• Shake-Shake regularization
• Deep Networks with Stochastic Depth
• Efficient Object Localization Using Convolutional Networks

Cited by in corpus (3)

• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning