A Survey of Label-noise Representation Learning: Past, Present and Future

Abstract

Classical machine learning implicitly assumes that labels of the training data are sampled from a clean distribution, which can be too restrictive for real-world scenarios. However, statistical-learning-based methods may not train deep learning models robustly with these noisy labels. Therefore, it is urgent to design Label-Noise Representation Learning (LNRL) methods for robustly training deep models with noisy labels. To fully understand LNRL, we conduct a survey study. We first clarify a formal definition for LNRL from the perspective of machine learning. Then, via the lens of learning theory and empirical study, we figure out why noisy labels affect deep models' performance. Based on the theoretical guidance, we categorize different LNRL methods into three directions. Under this unified taxonomy, we provide a thorough discussion of the pros and cons of different categories. More importantly, we summarize the essential components of robust LNRL, which can spark new directions. Lastly, we propose possible research directions within LNRL, such as new datasets, instance-dependent LNRL, and adversarial LNRL. We also envision potential directions beyond LNRL, such as learning with feature-noise, preference-noise, domain-noise, similarity-noise, graph-noise and demonstration-noise.

arXiv commentThe draft is kept updating; any comments and suggestions are welcome

References in corpus (21)

• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Classification with Noisy Labels by Importance Reweighting
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
• Unsupervised Label Noise Modeling and Loss Correction
• Strategies for Pre-training Graph Neural Networks
• How does Disagreement Help Generalization against Label Corruption?
• Positive-Unlabeled Learning with Non-Negative Risk Estimator
• Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
• Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels
• Learning from Complementary Labels
• A more robust boosting algorithm
• SIGUA: Forgetting May Make Learning with Noisy Labels More Robust
• Confidence Scores Make Instance-dependent Label-noise Learning Possible
• Maximum Entropy Semi-Supervised Inverse Reinforcement Learning
• Smooth Imitation Learning for Online Sequence Prediction
• Towards Robust ResNet: A Small Step but A Giant Leap
• A Second-Order Approach to Learning with Instance-Dependent Label Noise
• Butterfly: One-step Approach towards Wildly Unsupervised Domain Adaptation
• Multi-Class Classification from Noisy-Similarity-Labeled Data

Cited by in corpus (18)

• Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions
• Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
• Unsupervised Domain Adaptation of Black-Box Source Models
• Empirical investigation of multi-source cross-validation in clinical ECG classification
• Provably End-to-end Label-Noise Learning without Anchor Points
• Extended T: Learning with Mixed Closed-set and Open-set Noisy Labels
• Clusterability as an Alternative to Anchor Points When Learning with Noisy Labels
• Class Prior-Free Positive-Unlabeled Learning with Taylor Variational Loss for Hyperspectral Remote Sensing Imagery
• Learning From Long-Tailed Data With Noisy Labels
• A Second-Order Approach to Learning with Instance-Dependent Label Noise
• Weakly Supervised AUC Optimization: A Unified Partial AUC Approach
• Combining Self-Supervised and Supervised Learning with Noisy Labels
• PNT-Edge: Towards Robust Edge Detection with Noisy Labels by Learning Pixel-level Noise Transitions
• MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation
• Constrained Instance and Class Reweighting for Robust Learning under Label Noise
• Compensation Learning
• A Symmetric Loss Perspective of Reliable Machine Learning
• Knowledge Distillation with Noisy Labels for Natural Language Understanding