IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters

Abstract

In this work, we study robust deep learning against abnormal training data from the perspective of example weighting built in empirical loss functions, i.e., gradient magnitude with respect to logits, an angle that is not thoroughly studied so far. Consequently, we have two key findings: (1) Mean Absolute Error (MAE) Does Not Treat Examples Equally. We present new observations and insightful analysis about MAE, which is theoretically proved to be noise-robust. First, we reveal its underfitting problem in practice. Second, we analyse that MAE's noise-robustness is from emphasising on uncertain examples instead of treating training samples equally, as claimed in prior work. (2) The Variance of Gradient Magnitude Matters. We propose an effective and simple solution to enhance MAE's fitting ability while preserving its noise-robustness. Without changing MAE's overall weighting scheme, i.e., what examples get higher weights, we simply change its weighting variance non-linearly so that the impact ratio between two examples are adjusted. Our solution is termed Improved MAE (IMAE). We prove IMAE's effectiveness using extensive experiments: image classification under clean labels, synthetic label noise, and real-world unknown noise.

ICLR 2023, RTML Workshop paper. For the source code, based on the requests for academic research and kindness to cite our work, we will release and maintain it in https://github.com/XinshaoAmosWang/DeepCriticalLearning

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Learning to Reweight Examples for Robust Deep Learning
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• A Closer Look at Memorization in Deep Networks
• Dimensionality-Driven Learning with Noisy Labels
• Masking: A New Perspective of Noisy Supervision
• Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
• Truncated Cauchy Non-negative Matrix Factorization
• Joint Optimization Framework for Learning with Noisy Labels
• Smooth Loss Functions for Deep Top-k Classification
• Robust Inference via Generative Classifiers for Handling Noisy Labels
• Iterative Learning with Open-set Noisy Labels
• CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images
• Derivative Manipulation for General Example Weighting
• Deep Bilevel Learning
• ID-aware Quality for Set-based Person Re-identification

Cited by in corpus (14)

• Early-Learning Regularization Prevents Memorization of Noisy Labels
• LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environment
• Learning with Multiple Complementary Labels
• ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks
• Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Coresets for Robust Training of Neural Networks against Noisy Labels
• A Study on the Impact of Data Augmentation for Training Convolutional Neural Networks in the Presence of Noisy Labels
• Derivative Manipulation for General Example Weighting
• Confidence Adaptive Regularization for Deep Learning with Noisy Labels
• MetaInfoNet: Learning Task-Guided Information for Sample Reweighting
• A Survey on Deep Learning with Noisy Labels: How to train your model when you cannot trust on the annotations?
• Learning to Rectify for Robust Learning with Noisy Labels