Statistical Measures For Defining Curriculum Scoring Function
arXiv:2103.00147
Abstract
Curriculum learning is a training strategy that sorts the training examples by some measure of their difficulty and gradually exposes them to the learner to improve the network performance. Motivated by our insights from implicit curriculum ordering, we first introduce a simple curriculum learning strategy that uses statistical measures such as standard deviation and entropy values to score the difficulty of data points for real image classification tasks. We empirically show its improvements in performance with convolutional and fully-connected neural networks on multiple real image datasets. We also propose and study the performance of a dynamic curriculum learning algorithm. Our dynamic curriculum algorithm tries to reduce the distance between the network weight and an optimal weight at any training step by greedily sampling examples with gradients that are directed towards the optimal weight. Further, we use our algorithms to discuss why curriculum learning is helpful.
arXiv commentRevision 1: Corrected minor typos, added link to open-sourced codes, fixed Figures 7 as per reviews
References in corpus (9)
- On the Convergence of Adam and Beyond
- Improving Generalization Performance by Switching from Adam to SGD
- Online Batch Selection for Faster Training of Neural Networks
- Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks
- On The Power of Curriculum Learning in Training Deep Networks
- Curriculum Learning and Minibatch Bucketing in Neural Machine Translation
- An Empirical Exploration of Curriculum Learning for Neural Machine Translation
- Enhanced Directional Smoothing Algorithm for Edge-Preserving Smoothing of Synthetic-Aperture Radar Images
- Curriculum Audiovisual Learning