Multiplicative Reweighting for Robust Neural Network Optimization

Abstract

Neural networks are widespread due to their powerful performance. Yet, they degrade in the presence of noisy labels at training time. Inspired by the setting of learning with expert advice, where multiplicative weights (MW) updates were recently shown to be robust to moderate data corruptions in expert advice, we propose to use MW for reweighting examples during neural networks optimization. We theoretically establish the convergence of our method when used with gradient descent and prove its advantages in 1d cases. We then validate empirically our findings for the general case by showing that MW improves neural networks' accuracy in the presence of label noise on CIFAR-10, CIFAR-100 and Clothing1M. We also show the impact of our approach on adversarial robustness.

Our code is publicly available in https://github.com/NogaBar/mr_robust_optim

References in corpus (22)

• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Theoretically Principled Trade-off between Robustness and Accuracy
• Learning to Reweight Examples for Robust Deep Learning
• Classification with Noisy Labels by Importance Reweighting
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• WebVision Database: Visual Learning and Understanding from Web Data
• Dimensionality-Driven Learning with Noisy Labels
• Making Risk Minimization Tolerant to Label Noise
• Unsupervised Label Noise Modeling and Loss Correction
• Masking: A New Perspective of Noisy Supervision
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Symmetric Cross Entropy for Robust Learning with Noisy Labels
• Learning with Bad Training Data via Iterative Trimmed Loss Minimization
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Combating Label Noise in Deep Learning Using Abstention
• Joint Optimization Framework for Learning with Noisy Labels
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Learning from Noisy Labels with Distillation
• Classification vs regression in overparameterized regimes: Does the loss function matter?
• Statistical Learning with Conditional Value at Risk
• Learning compositional functions via multiplicative weight updates
• The Resistance to Label Noise in K-NN and DNN Depends on its Concentration

Cited by in corpus (1)

• Constrained Instance and Class Reweighting for Robust Learning under Label Noise