On The Power of Curriculum Learning in Training Deep Networks

Abstract

Training neural networks is traditionally done by providing a sequence of random mini-batches sampled uniformly from the entire training data. In this work, we analyze the effect of curriculum learning, which involves the non-uniform sampling of mini-batches, on the training of deep networks, and specifically CNNs trained for image recognition. To employ curriculum learning, the training algorithm must resolve 2 problems: (i) sort the training examples by difficulty; (ii) compute a series of mini-batches that exhibit an increasing level of difficulty. We address challenge (i) using two methods: transfer learning from some competitive ``teacher" network, and bootstrapping. In our empirical evaluation, both methods show similar benefits in terms of increased learning speed and improved final performance on test data. We address challenge (ii) by investigating different pacing functions to guide the sampling. The empirical investigation includes a variety of network architectures, using images from CIFAR-10, CIFAR-100 and subsets of ImageNet. We conclude with a novel theoretical analysis of curriculum learning, where we show how it effectively modifies the optimization landscape. We then define the concept of an ideal curriculum, and show that under mild conditions it does not change the corresponding global minimum of the optimization function.

arXiv commentIn proceedings, ICML 2019

Cited by in corpus (37)

• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
• Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
• Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
• DiverseDepth: Affine-invariant Depth Prediction Using Diverse Data
• Curriculum learning for improved femur fracture classification: scheduling data with prior knowledge and uncertainty
• ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization
• Deep Learning Through the Lens of Example Difficulty
• A Survey on Curriculum Learning
• Learning Adaptive Embedding Considering Incremental Class
• An Analytical Theory of Curriculum Learning in Teacher-Student Networks
• Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation
• Learning Fast Sample Re-weighting Without Reward Data
• Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification
• A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets
• Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Prediction
• Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking
• Focus-and-Expand: Training Guidance Through Gradual Manipulation of Input Features
• AdeNet: Deep learning architecture that identifies damaged electrical insulators in power lines
• CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability
• Adam with Bandit Sampling for Deep Learning
• Training With Data Dependent Dynamic Learning Rates
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Can we have it all? On the Trade-off between Spatial and Adversarial Robustness of Neural Networks
• Curriculum Learning with Diversity for Supervised Computer Vision Tasks
• Style Curriculum Learning for Robust Medical Image Segmentation
• Statistical Measures For Defining Curriculum Scoring Function
• Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning
• Which Samples Should be Learned First: Easy or Hard?
• Improving Generalization of Deep Fault Detection Models in the Presence of Mislabeled Data
• Competence-based Curriculum Learning for Multilingual Machine Translation
• Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning
• Dynamic Data Selection for Curriculum Learning via Ability Estimation
• Analyzing Curriculum Learning for Sentiment Analysis along Task Difficulty, Pacing and Visualization Axes