Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression
arXiv:2107.04497
Abstract
Heteroscedastic regression is the task of supervised learning where each label is subject to noise from a different distribution. This noise can be caused by the labelling process, and impacts negatively the performance of the learning algorithm as it violates the i.i.d. assumptions. In many situations however, the labelling process is able to estimate the variance of such distribution for each label, which can be used as an additional information to mitigate this impact. We adapt an inverse-variance weighted mean square error, based on the Gauss-Markov theorem, for parameter optimization on neural networks. We introduce Batch Inverse-Variance, a loss function which is robust to near-ground truth samples, and allows to control the effective learning rate. Our experimental results show that BIV improves significantly the performance of the networks on two noisy datasets, compared to L2 loss, inverse-variance weighting, as well as a filtering-based baseline.
arXiv commentAccepted at the Uncertainty in Deep Learning (UDL) workshop at ICML 2021
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Deep Residual Learning for Image Recognition
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
- Understanding deep learning requires rethinking generalization
- Classification with Noisy Labels by Importance Reweighting
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- Deep Learning is Robust to Massive Label Noise
- A Closer Look at Memorization in Deep Networks
- How does Disagreement Help Generalization against Label Corruption?
- Symmetric Cross Entropy for Robust Learning with Noisy Labels
- Learning with Bad Training Data via Iterative Trimmed Loss Minimization
- Probabilistic End-to-end Noise Correction for Learning with Noisy Labels
- Learning from Noisy Labels with Distillation
- Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization
- Deep Bilevel Learning