Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks

Abstract

We provide theoretical investigation of curriculum learning in the context of stochastic gradient descent when optimizing the convex linear regression loss. We prove that the rate of convergence of an ideal curriculum learning method is monotonically increasing with the difficulty of the examples. Moreover, among all equally difficult points, convergence is faster when using points which incur higher loss with respect to the current hypothesis. We then analyze curriculum learning in the context of training a CNN. We describe a method which infers the curriculum by way of transfer learning from another network, pre-trained on a different task. While this approach can only approximate the ideal curriculum, we observe empirically similar behavior to the one predicted by the theory, namely, a significant boost in convergence speed at the beginning of training. When the task is made more difficult, improvement in generalization performance is also observed. Finally, curriculum learning exhibits robustness against unfavorable conditions such as excessive regularization.

arXiv commentICML 2018

References in corpus (7)

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Theoretical Models of Learning to Learn
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Compressing Neural Networks with the Hashing Trick
• Learning to Execute
• Automated Curriculum Learning for Neural Networks
• Basic Level Categorization Facilitates Visual Object Recognition

Cited by in corpus (26)

• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
• DiverseDepth: Affine-invariant Depth Prediction Using Diverse Data
• Meta-Transfer Learning for Few-Shot Learning
• Curriculum learning for improved femur fracture classification: scheduling data with prior knowledge and uncertainty
• Online Deep Reinforcement Learning for Autonomous UAV Navigation and Exploration of Outdoor Environments
• Accuracy-based Curriculum Learning in Deep Reinforcement Learning
• SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
• A Survey on Curriculum Learning
• An Analytical Theory of Curriculum Learning in Teacher-Student Networks
• Curriculum based Dropout Discriminator for Domain Adaptation
• A Model of Two Tales: Dual Transfer Learning Framework for Improved Long-tail Item Recommendation
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• Theory of Curriculum Learning, with Convex Loss Functions
• Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Prediction
• Curriculum Design for Teaching via Demonstrations: Theory and Applications
• Difficulty Translation in Histopathology Images
• Can we have it all? On the Trade-off between Spatial and Adversarial Robustness of Neural Networks
• Statistical Measures For Defining Curriculum Scoring Function
• Goal-constrained Sparse Reinforcement Learning for End-to-End Driving
• Generalized Coarse-to-Fine Visual Recognition with Progressive Training
• An Empirical Comparison of Syllabuses for Curriculum Learning
• Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation
• Learning to Inpaint by Progressively Growing the Mask Regions
• Analyzing Curriculum Learning for Sentiment Analysis along Task Difficulty, Pacing and Visualization Axes