Faster Meta Update Strategy for Noise-Robust Deep Learning

Abstract

It has been shown that deep neural networks are prone to overfitting on biased training data. Towards addressing this issue, meta-learning employs a meta model for correcting the training bias. Despite the promising performances, super slow training is currently the bottleneck in the meta learning approaches. In this paper, we introduce a novel Faster Meta Update Strategy (FaMUS) to replace the most expensive step in the meta gradient computation with a faster layer-wise approximation. We empirically find that FaMUS yields not only a reasonably accurate but also a low-variance approximation of the meta gradient. We conduct extensive experiments to verify the proposed method on two tasks. We show our method is able to save two-thirds of the training time while still maintaining the comparable or achieving even better generalization performance. In particular, our method achieves the state-of-the-art performance on both synthetic and realistic noisy labels, and obtains promising performance on long-tailed recognition on standard benchmarks.

arXiv commentAccepted to CVPR 2021

References in corpus (15)

• Categorical Reparameterization with Gumbel-Softmax
• Learning to Reweight Examples for Robust Deep Learning
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• WebVision Database: Visual Learning and Understanding from Web Data
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Dimensionality-Driven Learning with Noisy Labels
• Unsupervised Label Noise Modeling and Loss Correction
• Are Anchor Points Really Indispensable in Label-Noise Learning?
• Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
• Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels
• Identifying Mislabeled Data using the Area Under the Margin Ranking
• Reducing Reparameterization Gradient Variance
• Meta Transition Adaptation for Robust Deep Learning with Noisy Labels
• Learning Soft Labels via Meta Learning
• SimAug: Learning Robust Representations from Simulation for Trajectory Prediction