A Simple Non-i.i.d. Sampling Approach for Efficient Training and Better Generalization

Abstract

While training on samples drawn from independent and identical distribution has been a de facto paradigm for optimizing image classification networks, humans learn new concepts in an easy-to-hard manner and on the selected examples progressively. Driven by this fact, we investigate the training paradigms where the samples are not drawn from independent and identical distribution. We propose a data sampling strategy, named Drop-and-Refresh (DaR), motivated by the learning behaviors of humans that selectively drop easy samples and refresh them only periodically. We show in our experiments that the proposed DaR strategy can maintain (and in many cases improve) the predictive accuracy even when the training cost is reduced by 15% on various datasets (CIFAR 10, CIFAR 100 and ImageNet) and with different backbone architectures (ResNets, DenseNets and MobileNets). Furthermore and perhaps more importantly, we find the ImageNet pre-trained models using our DaR sampling strategy achieves better transferability for the downstream tasks including object detection (+0.3 AP), instance segmentation (+0.3 AP), scene parsing (+0.5 mIoU) and human pose estimation (+0.6 AP). Our investigation encourages people to rethink the connections between the sampling strategy for training and the transferability of its learned features for pre-training ImageNet models.

arXiv commentTechnical report

References in corpus (11)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• mixup: Beyond Empirical Risk Minimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Mixed Precision Training
• Horovod: fast and easy distributed deep learning in TensorFlow
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Online Batch Selection for Faster Training of Neural Networks
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Decoupled Classification Refinement: Hard False Positive Suppression for Object Detection
• Safe Sample Screening for Support Vector Machines