Sequenced-Replacement Sampling for Deep Learning

Abstract

We propose sequenced-replacement sampling (SRS) for training deep neural networks. The basic idea is to assign a fixed sequence index to each sample in the dataset. Once a mini-batch is randomly drawn in each training iteration, we refill the original dataset by successively adding samples according to their sequence index. Thus we carry out replacement sampling but in a batched and sequenced way. In a sense, SRS could be viewed as a way of performing "mini-batch augmentation". It is particularly useful for a task where we have a relatively small images-per-class such as CIFAR-100. Together with a longer period of initial large learning rate, it significantly improves the classification accuracy in CIFAR-100 over the current state-of-the-art results. Our experiments indicate that training deeper networks with SRS is less prone to over-fitting. In the best case, we achieve an error rate as low as 10.10%.

References in corpus (12)

• Striving for Simplicity: The All Convolutional Net
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• FitNets: Hints for Thin Deep Nets
• Densely Connected Convolutional Networks
• Wide Residual Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Shake-Shake regularization
• Deep Networks with Stochastic Depth
• Practical recommendations for gradient-based training of deep architectures
• Why Random Reshuffling Beats Stochastic Gradient Descent
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Convergence Analysis of Distributed Stochastic Gradient Descent with Shuffling