Which Samples Should be Learned First: Easy or Hard?

Abstract

An effective weighting scheme for training samples is essential for learning tasks. Numerous weighting schemes have been proposed. Some schemes take the easy-first mode, whereas some others take the hard-first one. Naturally, an interesting yet realistic question is raised. Which samples should be learned first given a new learning task, easy or hard? To answer this question, both theoretical analyses and experimental verification are conducted. First, a general optimized objective function is proposed, revealing the relationship between the difficulty distribution and the difficulty-based sample weights. Second, on the basis of the optimized objective function, theoretical answers are obtained. Besides the easy-first and hard-first modes, there are two other priority modes, namely, medium-first and two-ends-first. The prior mode does not necessarily remain unchanged during the training process. Third, an effective and universal solution is proposed to select the optimal priority mode when there is no prior knowledge or theoretical clues. The four modes, namely, easy/medium/hard/two-ends-first, can be flexibly switched in the proposed solution. Fourth, a wide range of experiments is conducted under various scenarios to further compare the weighting schemes in different modes. On the basis of these works, reasonable and comprehensive answers are obtained. Factors including the distribution of samples' learning difficulties and the validation data determine which samples should be learned first in a learning task.

arXiv comment14 pages,10 figures

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• YOLOv4: Optimal Speed and Accuracy of Object Detection
• Rényi Divergence and Kullback-Leibler Divergence
• Learning to Reweight Examples for Robust Deep Learning
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• Pitfalls of Graph Neural Network Evaluation
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Denoising Implicit Feedback for Recommendation
• Dimensionality-Driven Learning with Noisy Labels
• Adaptive Sampling Towards Fast Graph Representation Learning
• On The Power of Curriculum Learning in Training Deep Networks
• Just Train Twice: Improving Group Robustness without Training Group Information
• Asymmetric Loss For Multi-Label Classification
• Stochastic Optimization with Importance Sampling
• Which Strategies Matter for Noisy Label Classification? Insight into Loss and Uncertainty