Mixing between the Cross Entropy and the Expectation Loss Terms

Abstract

The cross entropy loss is widely used due to its effectiveness and solid theoretical grounding. However, as training progresses, the loss tends to focus on hard to classify samples, which may prevent the network from obtaining gains in performance. While most work in the field suggest ways to classify hard negatives, we suggest to strategically leave hard negatives behind, in order to focus on misclassified samples with higher probabilities. We show that adding to the optimization goal the expectation loss, which is a better approximation of the zero-one loss, helps the network to achieve better accuracy. We, therefore, propose to shift between the two losses during training, focusing more on the expectation loss gradually during the later stages of training. Our experiments show that the new training protocol improves performance across a diverse set of classification domains, including computer vision, natural language processing, tabular data, and sequences. Our code and scripts are available at supplementary.

arXiv comment8 pages, 3 figures

References in corpus (16)

• Explaining and Harnessing Adversarial Examples
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Focal Loss for Dense Object Detection
• Random Erasing Data Augmentation
• Identity Mappings in Deep Residual Networks
• Robust Loss Functions under Label Noise for Deep Neural Networks
• Stochastic Gradient Descent as Approximate Bayesian Inference
• Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Extending Defensive Distillation
• On Loss Functions for Deep Neural Networks in Classification
• Learning From Noisy Large-Scale Datasets With Minimal Supervision
• Boosting Adversarial Attacks with Momentum
• Statistical Analysis on E-Commerce Reviews, with Sentiment Classification using Bidirectional Recurrent Neural Network (RNN)
• Taming the Cross Entropy Loss