Sigsoftmax: Reanalysis of the Softmax Bottleneck

Abstract

Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an output activation function for breaking the softmax bottleneck without additional parameters. We re-analyze the softmax bottleneck from the perspective of the output set of log-softmax and identify the cause of the softmax bottleneck. On the basis of this analysis, we propose sigsoftmax, which is composed of a multiplication of an exponential function and sigmoid function. Sigsoftmax can break the softmax bottleneck. The experiments on language modeling demonstrate that sigsoftmax and mixture of sigsoftmax outperform softmax and mixture of softmax, respectively.

arXiv comment15pages, 2 figures

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Sequence to Sequence Learning with Neural Networks
• On the difficulty of training Recurrent Neural Networks
• Recurrent Neural Network Regularization
• Pointer Sentinel Mixture Models
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family
• Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation

Cited by in corpus (7)

• Deep Complex Networks for Protocol-Agnostic Radio Frequency Device Fingerprinting in the Wild
• Constructive Type-Logical Supertagging with Self-Attention Networks
• Softmax-based Classification is k-means Clustering: Formal Proof, Consequences for Adversarial Attacks, and Improvement through Centroid Based Tailoring
• Extracting and Learning a Dependency-Enhanced Type Lexicon for Dutch
• Yet Another Representation of Binary Decision Trees: A Mathematical Demonstration
• Decision Machines: Congruent Decision Trees
• Recognition Awareness: An Application of Latent Cognizance to Open-Set Recognition