When Does Label Smoothing Help?

Abstract

The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing has been used in many state-of-the-art models, including image classification, language translation and speech recognition. Despite its widespread use, label smoothing is still poorly understood. Here we show empirically that in addition to improving generalization, label smoothing improves model calibration which can significantly improve beam-search. However, we also observe that if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective. To explain these observations, we visualize how label smoothing changes the representations learned by the penultimate layer of the network. We show that label smoothing encourages the representations of training examples from the same class to group in tight clusters. This results in loss of information in the logits about resemblances between instances of different classes, which is necessary for distillation, but does not hurt generalization or calibration of the model's predictions.

arXiv commentAccepted at NeurIPS 2019, corrected mutual information formulas

References in corpus (5)

• Distilling the Knowledge in a Neural Network
• On Calibration of Modern Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• Analyzing Uncertainty in Neural Machine Translation
• Calibration of Encoder Decoder Models for Neural Machine Translation

Cited by in corpus (115)

• Knowledge Distillation: A Survey
• A Survey on Data Augmentation for Text Classification
• Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification
• Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Series
• Active label cleaning for improved dataset quality under resource constraints
• Dynamic Object Aware LiDAR SLAM based on Automatic Generation of Training Data
• Understanding and Improving Knowledge Distillation
• 6GAN: IPv6 Multi-Pattern Target Generation via Generative Adversarial Nets with Reinforcement Learning
• Resolving Class Imbalance in Object Detection with Weighted Cross Entropy Losses
• Long-tail learning via logit adjustment
• Monitoring War Destruction from Space: A Machine Learning Approach
• Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
• Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding
• Detecting Outliers with Foreign Patch Interpolation
• Molecule Identification with Rotational Spectroscopy and Probabilistic Deep Learning
• Unbalanced minibatch Optimal Transport; applications to Domain Adaptation
• Learning from Noisy Labels for Entity-Centric Information Extraction
• The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions
• Towards contrast-agnostic soft segmentation of the spinal cord
• Regularization via Structural Label Smoothing
• Automated Human Cell Classification in Sparse Datasets using Few-Shot Learning
• NORESQA: A Framework for Speech Quality Assessment using Non-Matching References
• An Expandable Machine Learning-Optimization Framework to Sequential Decision-Making
• Probabilistic Deep Learning to Quantify Uncertainty in Air Quality Forecasting
• GaitFormer: Learning Gait Representations with Noisy Multi-Task Learning
• Mish: A Self Regularized Non-Monotonic Activation Function
• Learning Soft Labels via Meta Learning
• Is Your Model Sensitive? SPeDaC: A New Benchmark for Detecting and Classifying Sensitive Personal Data
• Connecting optical morphology, environment, and HI mass fraction for low-redshift galaxies using deep learning
• Does label smoothing mitigate label noise?
• Transformer with Selective Shuffled Position Embedding and Key-Patch Exchange Strategy for Early Detection of Knee Osteoarthritis
• Why distillation helps: a statistical perspective
• Learning Neural Network Subspaces
• Revisiting Knowledge Distillation via Label Smoothing Regularization
• From Universal Language Model to Downstream Task: Improving RoBERTa-Based Vietnamese Hate Speech Detection
• Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
• Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
• Fair Comparison: Quantifying Variance in Resultsfor Fine-grained Visual Categorization
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Distance-Based Learning from Errors for Confidence Calibration
• ConDA: Continual Unsupervised Domain Adaptation
• Neural Proof Nets
• VIPriors 1: Visual Inductive Priors for Data-Efficient Deep Learning Challenges
• Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models
• LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training
• Machine Learning for Cataract Classification and Grading on Ophthalmic Imaging Modalities: A Survey
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Defending Against Adversarial Attacks by Suppressing the Largest Eigenvalue of Fisher Information Matrix
• Generation, augmentation, and alignment: A pseudo-source domain based method for source-free domain adaptation
• Smoothed Contrastive Learning for Unsupervised Sentence Embedding
• Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition
• Soft Calibration Objectives for Neural Networks
• Automated Learning Rate Scheduler for Large-batch Training
• A Challenging Benchmark of Anime Style Recognition
• Facing the Hard Problems in FGVC
• Nearest Neighborhood-Based Deep Clustering for Source Data-absent Unsupervised Domain Adaptation
• Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation
• On the Inference Calibration of Neural Machine Translation
• Semantic-embedded Similarity Prototype for Scene Recognition
• E-Stitchup: Data Augmentation for Pre-Trained Embeddings
• On the Effectiveness of Regularization Against Membership Inference Attacks
• The 2021 Hotel-ID to Combat Human Trafficking Competition Dataset
• ReSmooth: Detecting and Utilizing OOD Samples when Training with Data Augmentation
• Masked Pre-trained Encoder base on Joint CTC-Transformer
• An Investigation of the Effectiveness of Phase for Audio Classification
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• COVID Detection in Chest CTs: Improving the Baseline on COV19-CT-DB
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts
• Spectrogram Inpainting for Interactive Generation of Instrument Sounds
• SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations
• An Exploratory Study on Perceptual Spaces of the Singing Voice
• Generating Out of Distribution Adversarial Attack using Latent Space Poisoning
• Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding
• Multiclass Spinal Cord Tumor Segmentation on MRI with Deep Learning
• Double Backpropagation for Training Autoencoders against Adversarial Attack
• Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations
• Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability
• HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection
• Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization
• Accent Estimation of Japanese Words from Their Surfaces and Romanizations for Building Large Vocabulary Accent Dictionaries
• Structure-Aware Label Smoothing for Graph Neural Networks
• Mapping Housing Stock Characteristics from Drone Images for Climate Resilience in the Caribbean
• Regularized Evolutionary Population-Based Training
• 6VecLM: Language Modeling in Vector Space for IPv6 Target Generation
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• Sample Efficient Learning of Image-Based Diagnostic Classifiers Using Probabilistic Labels
• A New Training Framework for Deep Neural Network
• Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification
• Diversifying Dialog Generation via Adaptive Label Smoothing
• 2nd Place Solution to Google Landmark Retrieval 2021
• Instance-Dependent Partial Label Learning
• Collaborative Semantic Aggregation and Calibration for Federated Domain Generalization
• CounterNet: End-to-End Training of Prediction Aware Counterfactual Explanations
• Detecting Escalation Level from Speech with Transfer Learning and Acoustic-Lexical Information Fusion
• Spending Your Winning Lottery Better After Drawing It
• Towards Hardware-Agnostic Gaze-Trackers
• Matrix Smoothing: A Regularization for DNN with Transition Matrix under Noisy Labels
• SIPA: A Simple Framework for Efficient Networks
• Learning and Exploiting Interclass Visual Correlations for Medical Image Classification
• Introspective Learning by Distilling Knowledge from Online Self-explanation
• Regularization via Adaptive Pairwise Label Smoothing
• Learning and Reasoning with the Graph Structure Representation in Robotic Surgery
• MadNet: Using a MAD Optimization for Defending Against Adversarial Attacks
• Credal Self-Supervised Learning
• Less is More: Feature Selection for Adversarial Robustness with Compressive Counter-Adversarial Attacks
• Calibrating the Full Predictive Class Distribution of 3D Object Detectors for Autonomous Driving
• ReINTEL Challenge 2020: A Comparative Study of Hybrid Deep Neural Network for Reliable Intelligence Identification on Vietnamese SNSs
• On-target Adaptation
• Mutual Teaching for Graph Convolutional Networks
• Midpoint Regularization: from High Uncertainty Training to Conservative Classification
• Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation
• Exploiting Class Similarity for Machine Learning with Confidence Labels and Projective Loss Functions
• Learning from Matured Dumb Teacher for Fine Generalization
• Chi-square Loss for Softmax: an Echo of Neural Network Structure
• Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection