On the Robustness of Monte Carlo Dropout Trained with Noisy Labels
arXiv:2103.12002
Abstract
The memorization effect of deep learning hinders its performance to effectively generalize on test set when learning with noisy labels. Prior study has discovered that epistemic uncertainty techniques are robust when trained with noisy labels compared with neural networks without uncertainty estimation. They obtain prolonged memorization effect and better generalization performance under the adversarial setting of noisy labels. Due to its superior performance amongst other selected epistemic uncertainty methods under noisy labels, we focus on Monte Carlo Dropout (MCDropout) and investigate why it is robust when trained with noisy labels. Through empirical studies on datasets MNIST, CIFAR-10, Animal-10n, we deep dive into three aspects of MCDropout under noisy label setting: 1. efficacy: understanding the learning behavior and test accuracy of MCDropout when training set contains artificially generated or naturally embedded label noise; 2. representation volatility: studying the responsiveness of neurons by examining the mean and standard deviation on each neuron's activation; 3. network sparsity: investigating the network support of MCDropout in comparison with deterministic neural networks. Our findings suggest that MCDropout further sparsifies and regularizes the deterministic neural networks and thus provides higher robustness against noisy labels.
References in corpus (17)
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
- Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels
- The State of Sparsity in Deep Neural Networks
- GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
- How does Disagreement Help Generalization against Label Corruption?
- Robust Loss Functions under Label Noise for Deep Neural Networks
- Symmetric Cross Entropy for Robust Learning with Noisy Labels
- Decoupling "when to update" from "how to update"
- Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach
- Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
- Curriculum Loss: Robust Learning and Generalization against Label Corruption
- Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
- SELF: Learning to Filter Noisy Labels with Self-Ensembling
- Label Noise Types and Their Effects on Deep Learning
- Training Sparse Neural Networks
- Robust Deep Learning with Active Noise Cancellation for Spatial Computing