Locally Adaptive Label Smoothing for Predictive Churn
arXiv:2102.05140
Abstract
Training modern neural networks is an inherently noisy process that can lead to high \emph{prediction churn} -- disagreements between re-trainings of the same model due to factors such as randomization in the parameter initialization and mini-batches -- even when the trained models all attain similar accuracies. Such prediction churn can be very undesirable in practice. In this paper, we present several baselines for reducing churn and show that training on soft labels obtained by adaptively smoothing each example's label based on the example's neighboring labels often outperforms the baselines on churn while improving accuracy on a variety of benchmark classification tasks and model architectures.
arXiv commentICML 2021
References in corpus (13)
- mixup: Beyond Empirical Risk Minimization
- Knowledge Distillation by On-the-Fly Native Ensemble
- Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
- Deep Ensembles: A Loss Landscape Perspective
- To Trust Or Not To Trust A Classifier
- How Does Batch Normalization Help Optimization?
- Online Batch Selection for Faster Training of Neural Networks
- Large scale distributed neural network training through online distillation
- Satisfying Real-world Goals with Dataset Constraints
- Combating Label Noise in Deep Learning Using Abstention
- Rates of Convergence for Nearest Neighbor Classification
- Fast Rates for a kNN Classifier Robust to Unknown Asymmetric Label Noise
- Adaptive Hausdorff estimation of density level sets