Unbiased scalable softmax optimization

Abstract

Recent neural network and language models rely on softmax distributions with an extremely large number of categories. Since calculating the softmax normalizing constant in this context is prohibitively expensive, there is a growing literature of efficiently computable but biased estimates of the softmax. In this paper we propose the first unbiased algorithms for maximizing the softmax likelihood whose work per iteration is independent of the number of classes and datapoints (and no extra work is required at the end of each epoch). We show that our proposed unbiased methods comprehensively outperform the state-of-the-art on seven real world datasets.

References in corpus (6)

• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• LSHTC: A Benchmark for Large-Scale Text Classification
• An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family
• Logarithmic Time One-Against-Some
• Simultaneous Learning of Trees and Representations for Extreme Classification and Density Estimation
• DS-MLR: Exploiting Double Separability for Scaling up Distributed Multinomial Logistic Regression

Cited by in corpus (3)

• Augment and Reduce: Stochastic Inference for Large Categorical Distributions
• Sampled Softmax with Random Fourier Features
• Compositional Stochastic Average Gradient for Machine Learning and Related Applications