BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies

Abstract

We propose BlackOut, an approximation algorithm to efficiently train massive recurrent neural network language models (RNNLMs) with million word vocabularies. BlackOut is motivated by using a discriminative loss, and we describe a new sampling strategy which significantly reduces computation while improving stability, sample efficiency, and rate of convergence. One way to understand BlackOut is to view it as an extension of the DropOut strategy to the output layer, wherein we use a discriminative training loss and a weighted sampling scheme. We also establish close connections between BlackOut, importance sampling, and noise contrastive estimation (NCE). Our experiments, on the recently released one billion word language modeling benchmark, demonstrate scalability and accuracy of BlackOut; we outperform the state-of-the art, and achieve the lowest perplexity scores on this dataset. Moreover, unlike other established methods which typically require GPUs or CPU clusters, we show that a carefully implemented version of BlackOut requires only 1-10 days on a single machine to train a RNNLM with a million word vocabulary and billions of parameters on one billion words. Although we describe BlackOut in the context of RNNLM training, it can be used to any networks with large softmax output layers.

Published as a conference paper at ICLR 2016

References in corpus (3)

• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)
• Deep Networks With Large Output Spaces

Cited by in corpus (8)

• Language Modeling with Gated Convolutional Networks
• Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
• Efficient softmax approximation for GPUs
• One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities
• Automatic Speech Recognition with Very Large Conversational Finnish and Estonian Vocabularies
• TAPAS: Two-pass Approximate Adaptive Sampling for Softmax
• Candidates vs. Noises Estimation for Large Multi-Class Classification Problem
• A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection