Robust Training with Ensemble Consensus
arXiv:1910.09792
Abstract
Since deep neural networks are over-parameterized, they can memorize noisy examples. We address such a memorization issue in the presence of label noise. From the fact that deep neural networks cannot generalize to neighborhoods of memorized features, we hypothesize that noisy examples do not consistently incur small losses on the network under a certain perturbation. Based on this, we propose a novel training method called Learning with Ensemble Consensus (LEC) that prevents overfitting to noisy examples by removing them based on the consensus of an ensemble of perturbed networks. One of the proposed LECs, LTEC outperforms the current state-of-the-art methods on noisy MNIST, CIFAR-10, and CIFAR-100 in an efficient manner.
arXiv commentICLR 2020
References in corpus (11)
- Training Deep Neural Networks on Noisy Labels with Bootstrapping
- Training Convolutional Networks with Noisy Labels
- Using Pre-Training Can Improve Model Robustness and Uncertainty
- Deep Learning is Robust to Massive Label Noise
- A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
- A Closer Look at Memorization in Deep Networks
- WebVision Database: Visual Learning and Understanding from Web Data
- Dimensionality-Driven Learning with Noisy Labels
- Learning with Bad Training Data via Iterative Trimmed Loss Minimization
- An Empirical Study of Example Forgetting during Deep Neural Network Learning
- Controlling Over-generalization and its Effect on Adversarial Examples Generation and Detection