Regularizing Neural Networks by Penalizing Confident Output Distributions

Abstract

We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT'14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.

arXiv commentSubmitted to ICLR 2017

Cited by in corpus (127)

• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• Beyond English-Centric Multilingual Machine Translation
• Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
• Pathologies of Neural Models Make Interpretations Difficult
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Fantastic Generalization Measures and Where to Find Them
• Large scale distributed neural network training through online distillation
• A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU
• Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020
• Abstractive Summarization of Reddit Posts with Multi-level Memory Networks
• Excessive Invariance Causes Adversarial Vulnerability
• Training Neural Response Selection for Task-Oriented Dialogue Systems
• Parameters Estimation for the Cosmic Microwave Background with Bayesian Neural Networks
• Self-Distillation as Instance-Specific Label Smoothing
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
• Calibration of Encoder Decoder Models for Neural Machine Translation
• Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network
• Training Binary Neural Networks through Learning with Noisy Supervision
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• Confidence Regularized Self-Training
• Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
• Controlling Information Capacity of Binary Neural Network
• Regularizing Class-wise Predictions via Self-knowledge Distillation
• Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification
• Entropic Out-of-Distribution Detection: Seamless Detection of Unknown Examples
• Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
• Automatic segmentation method of pelvic floor levator hiatus in ultrasound using a self-normalising neural network
• BERT & Family Eat Word Salad: Experiments with Text Understanding
• Pre-trained Language Model Representations for Language Generation
• On Using SpecAugment for End-to-End Speech Translation
• Learning Optimal Representations with the Decodable Information Bottleneck
• Learning Soft Labels via Meta Learning
• Does label smoothing mitigate label noise?
• Flexible Dataset Distillation: Learn Labels Instead of Images
• Towards Noise-resistant Object Detection with Noisy Annotations
• Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
• ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks
• A Variational Dirichlet Framework for Out-of-Distribution Detection
• Temperature check: theory and practice for training models with softmax-cross-entropy losses
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
• The DKU-DukeECE System for the Self-Supervision Speaker Verification Task of the 2021 VoxCeleb Speaker Recognition Challenge
• Label Smoothing and Adversarial Robustness
• Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding
• Improving Back-Translation with Uncertainty-based Confidence Estimation
• An Investigation of how Label Smoothing Affects Generalization
• Stylistic Dialogue Generation via Information-Guided Reinforcement Learning Strategy
• Confidence Adaptive Regularization for Deep Learning with Noisy Labels
• Improved Adversarial Robustness via Logit Regularization Methods
• Measuring Dependence with Matrix-based Entropy Functional
• Approximating Instance-Dependent Noise via Instance-Confidence Embedding
• Beyond BLEU: Training Neural Machine Translation with Semantic Similarity
• Imbalanced Image Classification with Complement Cross Entropy
• Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach
• Optimal Completion Distillation for Sequence Learning
• Applying a Pre-trained Language Model to Spanish Twitter Humor Prediction
• Instance Cross Entropy for Deep Metric Learning
• Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms
• Calibration of Neural Networks using Splines
• On the Importance of Firth Bias Reduction in Few-Shot Classification
• Graph Mixture Density Networks
• Meta Dropout: Learning to Perturb Features for Generalization
• Hierarchical Expert Networks for Meta-Learning
• Deep Probability Estimation
• An iterative framework for self-supervised deep speaker representation learning
• Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization
• Scalable Syntax-Aware Language Models Using Knowledge Distillation
• Learning Class Unique Features in Fine-Grained Visual Classification
• Study Group Learning: Improving Retinal Vessel Segmentation Trained with Noisy Labels
• DcardNet: Diabetic Retinopathy Classification at Multiple Levels Based on Structural and Angiographic Optical Coherence Tomography
• Bridging In- and Out-of-distribution Samples for Their Better Discriminability
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration
• Neural Machine Translation: A Review and Survey
• SuperNet -- An efficient method of neural networks ensembling
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
• Learning Non-Monotonic Automatic Post-Editing of Translations from Human Orderings
• Non-Parametric Calibration for Classification
• Extending Recurrent Neural Aligner for Streaming End-to-End Speech Recognition in Mandarin
• Mixture Representation Learning with Coupled Autoencoders
• Adversarial Distillation for Ordered Top-k Attacks
• Probabilistic Approach for Road-Users Detection
• Probabilistic Object Classification using CNN ML-MAP layers
• Robust Question Answering Through Sub-part Alignment
• Decoupled Gradient Harmonized Detector for Partial Annotation: Application to Signet Ring Cell Detection
• Maximum Entropy Regularization and Chinese Text Recognition
• Introspective Learning by Distilling Knowledge from Online Self-explanation
• Stronger Transformers for Neural Multi-Hop Question Generation
• Why Do Better Loss Functions Lead to Less Transferable Features?
• Privileged Features Distillation at Taobao Recommendations
• Knowledge Transfer Graph for Deep Collaborative Learning
• Dynamic Slimmable Network
• Semi-Supervised Text Classification via Self-Pretraining
• Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP
• Identifying and Exploiting Structures for Reliable Deep Learning
• Midpoint Regularization: from High Uncertainty Training to Conservative Classification
• Truth Discovery in Sequence Labels from Crowds
• STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
• On Deep Neural Network Calibration by Regularization and its Impact on Refinement
• Rethinking Uncertainty in Deep Learning: Whether and How it Improves Robustness
• Joint Text and Label Generation for Spoken Language Understanding
• Class Interference Regularization
• Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing
• Regularization via Adaptive Pairwise Label Smoothing
• Deep Unsupervised Image Anomaly Detection: An Information Theoretic Framework
• IIE-NLP-Eyas at SemEval-2021 Task 4: Enhancing PLM for ReCAM with Special Tokens, Re-Ranking, Siamese Encoders and Back Translation
• A practical two-stage training strategy for multi-stream end-to-end speech recognition
• Genome Variant Calling with a Deep Averaging Network
• Classification as Decoder: Trading Flexibility for Control in Medical Dialogue
• Integrating Knowledge into End-to-End Speech Recognition from External Text-Only Data
• Unsupervised Domain Adaptive Object Detection using Forward-Backward Cyclic Adaptation
• Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition
• Self-Teaching Networks
• Friends and Foes in Learning from Noisy Labels
• ReMix: Towards Image-to-Image Translation with Limited Data
• Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search
• The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks
• AugLabel: Exploiting Word Representations to Augment Labels for Face Attribute Classification
• A Simple and Effective Approach to Automatic Post-Editing with Transfer Learning
• Homography augumented momentum constrastive learning for SAR image retrieval