Towards Understanding Label Smoothing

Abstract

Label smoothing regularization (LSR) has a great success in training deep neural networks by stochastic algorithms such as stochastic gradient descent and its variants. However, the theoretical understanding of its power from the view of optimization is still rare. This study opens the door to a deep understanding of LSR by initiating the analysis. In this paper, we analyze the convergence behaviors of stochastic gradient descent with label smoothing regularization for solving non-convex problems and show that an appropriate LSR can help to speed up the convergence by reducing the variance. More interestingly, we proposed a simple yet effective strategy, namely Two-Stage LAbel smoothing algorithm (TSLA), that uses LSR in the early training epochs and drops it off in the later training epochs. We observe from the improved convergence result of TSLA that it benefits from LSR in the first stage and essentially converges faster in the second stage. To the best of our knowledge, this is the first work for understanding the power of LSR via establishing convergence complexity of stochastic methods with LSR in non-convex optimization. We empirically demonstrate the effectiveness of the proposed method in comparison with baselines on training ResNet models over benchmark data sets.

References in corpus (18)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Wide Residual Networks
• On the Convergence of Adam and Beyond
• A Convergence Theory for Deep Learning via Over-Parameterization
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Improved training of end-to-end attention models for speech recognition
• Label Refinery: Improving ImageNet Classification through Label Progression
• Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
• Adaptive Regularization of Labels
• Does label smoothing mitigate label noise?
• Revisiting Knowledge Distillation via Label Smoothing Regularization
• A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
• Adversarial Robustness via Label-Smoothing
• Defending Against Adversarial Attacks by Suppressing the Largest Eigenvalue of Fisher Information Matrix
• On the Inference Calibration of Neural Machine Translation

Cited by in corpus (5)

• Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks
• WeMix: How to Better Utilize Data Augmentation
• Extending Label Smoothing Regularization with Self-Knowledge Distillation
• Why Do Better Loss Functions Lead to Less Transferable Features?
• Jitter: Random Jittering Loss Function