Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization
arXiv:2006.15766
Abstract
Real-world large-scale datasets are heteroskedastic and imbalanced -- labels have varying levels of uncertainty and label distributions are long-tailed. Heteroskedasticity and imbalance challenge deep learning algorithms due to the difficulty of distinguishing among mislabeled, ambiguous, and rare examples. Addressing heteroskedasticity and imbalance simultaneously is under-explored. We propose a data-dependent regularization technique for heteroskedastic datasets that regularizes different regions of the input space differently. Inspired by the theoretical derivation of the optimal regularization strength in a one-dimensional nonparametric classification setting, our approach adaptively regularizes the data points in higher-uncertainty, lower-density regions more heavily. We test our method on several benchmark tasks, including a real-world heteroskedastic and imbalanced dataset, WebVision. Our experiments corroborate our theory and demonstrate a significant improvement over other methods in noise-robust deep learning.
arXiv commentto appear in ICLR 2021
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Understanding deep learning requires rethinking generalization
- Learning to Reweight Examples for Robust Deep Learning
- Classification with Noisy Labels by Importance Reweighting
- A Closer Look at Memorization in Deep Networks
- WebVision Database: Visual Learning and Understanding from Web Data
- Robust Large Margin Deep Neural Networks
- Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
- The Role of Neural Network Activation Functions
- On The Power of Curriculum Learning in Training Deep Networks
- Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels
- Local and global asymptotic inference in smoothing spline models
- Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
- Learning with Bounded Instance- and Label-dependent Label Noise
- Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
- Learning from Binary Labels with Instance-Dependent Corruption
- How do infinite width bounded norm networks look in function space?
- Coresets for Robust Training of Neural Networks against Noisy Labels
- Robust and On-the-fly Dataset Denoising for Image Classification