An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family

Abstract

In a multi-class classification problem, it is standard to model the output of a neural network as a categorical distribution conditioned on the inputs. The output must therefore be positive and sum to one, which is traditionally enforced by a softmax. This probabilistic mapping allows to use the maximum likelihood principle, which leads to the well-known log-softmax loss. However the choice of the softmax function seems somehow arbitrary as there are many other possible normalizing functions. It is thus unclear why the log-softmax loss would perform better than other loss alternatives. In particular Vincent et al. (2015) recently introduced a class of loss functions, called the spherical family, for which there exists an efficient algorithm to compute the updates of the output weights irrespective of the output size. In this paper, we explore several loss functions from this family as possible alternatives to the traditional log-softmax. In particular, we focus our investigation on spherical bounds of the log-softmax loss and on two spherical log-likelihood losses, namely the log-Spherical Softmax suggested by Vincent et al. (2015) and the log-Taylor Softmax that we introduce. Although these alternatives do not yield as good results as the log-softmax loss on two language modeling tasks, they surprisingly outperform it in our experiments on MNIST and CIFAR-10, suggesting that they might be relevant in a broad range of applications.

Published at ICLR 2016

Cited by in corpus (11)

• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Sigsoftmax: Reanalysis of the Softmax Bottleneck
• Simultaneous Learning of Trees and Representations for Extreme Classification and Density Estimation
• Unbiased scalable softmax optimization
• Efficient Training on Very Large Corpora via Gramian Estimation
• The Z-loss: a shift and scale invariant classification loss belonging to the Spherical Family
• Augment and Reduce: Stochastic Inference for Large Categorical Distributions
• Robust Propensity Score Computation Method based on Machine Learning with Label-corrupted Data
• Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
• Sampled Softmax with Random Fourier Features
• Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via GDPA Linearization