Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation

Abstract

Over the past few years, softmax and SGD have become a commonly used component and the default training strategy in CNN frameworks, respectively. However, when optimizing CNNs with SGD, the saturation behavior behind softmax always gives us an illusion of training well and then is omitted. In this paper, we first emphasize that the early saturation behavior of softmax will impede the exploration of SGD, which sometimes is a reason for model converging at a bad local-minima, then propose Noisy Softmax to mitigating this early saturation issue by injecting annealed noise in softmax during each iteration. This operation based on noise injection aims at postponing the early saturation and further bringing continuous gradients propagation so as to significantly encourage SGD solver to be more exploratory and help to find a better local-minima. This paper empirically verifies the superiority of the early softmax desaturation, and our method indeed improves the generalization ability of CNN model by regularization. We experimentally find that this early desaturation helps optimization in many tasks, yielding state-of-the-art or competitive results on several popular benchmark datasets.

arXiv comment10 pages, 7 figures, CVPR2017

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Densely Connected Convolutional Networks
• Learning Face Representation from Scratch
• Weight Uncertainty in Neural Networks
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Large-Margin Softmax Loss for Convolutional Neural Networks
• Deep Networks with Stochastic Depth
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Mask-CNN: Localizing Parts and Selecting Descriptors for Fine-Grained Image Recognition
• Noisy Activation Functions
• Estimating or Propagating Gradients Through Stochastic Neurons

Cited by in corpus (10)

• Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments
• Minimum Margin Loss for Deep Face Recognition
• Rethinking Feature Distribution for Loss Functions in Image Classification
• Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation
• Sigsoftmax: Reanalysis of the Softmax Bottleneck
• Energy Confused Adversarial Metric Learning for Zero-Shot Image Retrieval and Clustering
• GB-CosFace: Rethinking Softmax-based Face Recognition from the Perspective of Open Set Classification
• Introducing Noise in Decentralized Training of Neural Networks
• Tackling Early Sparse Gradients in Softmax Activation Using Leaky Squared Euclidean Distance
• Center Loss Regularization for Continual Learning