Midpoint Regularization: from High Uncertainty Training to Conservative Classification

Abstract

Label Smoothing (LS) improves model generalization through penalizing models from generating overconfident output distributions. For each training sample the LS strategy smooths the one-hot encoded training signal by distributing its distribution mass over the non-ground truth classes. We extend this technique by considering example pairs, coined PLS. PLS first creates midpoint samples by averaging random sample pairs and then learns a smoothing distribution during training for each of these midpoint samples, resulting in midpoints with high uncertainty labels for training. We empirically show that PLS significantly outperforms LS, achieving up to 30% of relative classification error reduction. We also visualize that PLS produces very low winning softmax scores for both in and out of distribution samples.

Accepted to ECML-PKDD 2021. arXiv admin note: substantial text overlap with arXiv:2012.01559

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Augmenting Data with Mixup for Sentence Classification: An Empirical Study
• Variational Information Distillation for Knowledge Transfer
• Does label smoothing mitigate label noise?
• MixUp as Directional Adversarial Training