Deep Networks With Large Output Spaces

Abstract

Deep neural networks have been extremely successful at various image, speech, video recognition tasks because of their ability to model deep structures within the data. However, they are still prohibitively expensive to train and apply for problems containing millions of classes in the output layer. Based on the observation that the key computation common to most neural network layers is a vector/matrix product, we propose a fast locality-sensitive hashing technique to approximate the actual dot product enabling us to scale up the training and inference to millions of output classes. We evaluate our technique on three diverse large-scale recognition tasks and show that our approach can train large-scale models at a faster rate (in terms of steps/total time) compared to baseline methods.

Cited by in corpus (15)

• Beyond One-hot Encoding: lower dimensional target embedding
• Hierarchical Memory Networks
• One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities
• BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies
• Strategies for Training Large Vocabulary Neural Language Models
• Clustering is Efficient for Approximate Maximum Inner Product Search
• Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targets
• Fast Amortized Inference and Learning in Log-linear Models with Randomly Perturbed Nearest Neighbor Search
• Augment and Reduce: Stochastic Inference for Large Categorical Distributions
• Compressing Gradient Optimizers via Count-Sketches
• ImageNet-21K Pretraining for the Masses
• Reference Product Search
• Sampled Softmax with Random Fourier Features
• Extreme Classification in Log Memory
• Exact gradient updates in time independent of output size for the spherical loss family