Dimensionality-Driven Learning with Noisy Labels

Abstract

Datasets with significant proportions of noisy (incorrect) class labels present challenges for training accurate Deep Neural Networks (DNNs). We propose a new perspective for understanding DNN generalization for such datasets, by investigating the dimensionality of the deep representation subspace of training samples. We show that from a dimensionality perspective, DNNs exhibit quite distinctive learning styles when trained with clean labels versus when trained with a proportion of noisy labels. Based on this finding, we develop a new dimensionality-driven learning strategy, which monitors the dimensionality of subspaces during training and adapts the loss function accordingly. We empirically demonstrate that our approach is highly tolerant to significant proportions of noisy labels, and can effectively learn low-dimensional local subspaces that capture the data distribution.

arXiv commentIn Proceedings of the International Conference on Machine Learning (ICML), 2018

References in corpus (11)

• Opening the Black Box of Deep Neural Networks via Information
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Training Convolutional Networks with Noisy Labels
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• A Closer Look at Memorization in Deep Networks
• Deep Networks with Stochastic Depth
• Dimensionality-Driven Learning with Noisy Labels
• Learning from Noisy Labels with Distillation
• Learning From Noisy Large-Scale Datasets With Minimal Supervision
• Iterative Learning with Open-set Noisy Labels

Cited by in corpus (54)

• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels
• Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting
• Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
• Early-Learning Regularization Prevents Memorization of Noisy Labels
• Dimensionality-Driven Learning with Noisy Labels
• Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis
• Are Anchor Points Really Indispensable in Label-Noise Learning?
• Intrinsic dimension of data representations in deep neural networks
• Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression
• Curriculum Loss: Robust Learning and Generalization against Label Corruption
• LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environment
• Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning
• Understanding and Improving Early Stopping for Learning with Noisy Labels
• Identifying Mislabeled Data using the Area Under the Margin Ranking
• IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters
• Label Noise Types and Their Effects on Deep Learning
• Confidence Scores Make Instance-dependent Label-noise Learning Possible
• L_DMI: An Information-theoretic Noise-robust Loss Function
• What Do Neural Networks Learn When Trained With Random Labels?
• Learning with Bounded Instance- and Label-dependent Label Noise
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Provably End-to-end Label-Noise Learning without Anchor Points
• Uncertainty Based Detection and Relabeling of Noisy Image Labels
• MetaLabelNet: Learning to Generate Soft-Labels from Noisy-Labels
• Self-supervised learning for autonomous vehicles perception: A conciliation between analytical and learning methods
• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels
• FedDebug: Systematic Debugging for Federated Learning Applications
• ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks
• Derivative Manipulation for General Example Weighting
• Robust Training with Ensemble Consensus
• Faster Meta Update Strategy for Noise-Robust Deep Learning
• Boosting Co-teaching with Compression Regularization for Label Noise
• Countering Noisy Labels By Learning From Auxiliary Clean Labels
• Adaptive Sample Selection for Robust Learning under Label Noise
• Improving Generalization by Controlling Label-Noise Information in Neural Network Weights
• The Resistance to Label Noise in K-NN and DNN Depends on its Concentration
• Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks
• MetaInfoNet: Learning Task-Guided Information for Sample Reweighting
• Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection
• Learning to Combat Noisy Labels via Classification Margins
• NLNL: Negative Learning for Noisy Labels
• Joint Negative and Positive Learning for Noisy Labels
• Facial Emotion Recognition with Noisy Multi-task Annotations
• Co-Seg: An Image Segmentation Framework Against Label Corruption
• Matrix Smoothing: A Regularization for DNN with Transition Matrix under Noisy Labels
• Multiplicative Reweighting for Robust Neural Network Optimization
• Compensation Learning
• Improving Generalization of Deep Fault Detection Models in the Presence of Mislabeled Data
• Learning with Noisy Labels for Sentence-level Sentiment Classification
• Improved Mix-up with KL-Entropy for Learning From Noisy Labels
• Learning advisor networks for noisy image classification
• Which Samples Should be Learned First: Easy or Hard?
• A robust approach for deep neural networks in presence of label noise: relabelling and filtering instances during training