A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

Abstract

In spite of their superior performance, neural probabilistic language models (NPLMs) remain far less widely used than n-gram models due to their notoriously long training times, which are measured in weeks even for moderately-sized datasets. Training NPLMs is computationally expensive because they are explicitly normalized, which leads to having to consider all words in the vocabulary when computing the log-likelihood gradients. We propose a fast and simple algorithm for training NPLMs based on noise-contrastive estimation, a newly introduced procedure for estimating unnormalized continuous distributions. We investigate the behaviour of the algorithm on the Penn Treebank corpus and show that it reduces the training times by more than an order of magnitude without affecting the quality of the resulting models. The algorithm is also more efficient and much more stable than importance sampling because it requires far fewer noise samples to perform well. We demonstrate the scalability of the proposed approach by training several neural language models on a 47M-word corpus with a 80K-word vocabulary, obtaining state-of-the-art results on the Microsoft Research Sentence Completion Challenge dataset.

Appears in Proceedings of the 29th International Conference on Machine Learning (ICML 2012)

Cited by in corpus (56)

• Distributed Representations of Words and Phrases and their Compositionality
• Contrastive Representation Learning: A Framework and Review
• BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
• VERSE: Versatile Graph Embeddings from Similarity Measures
• "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
• A Survey Of Cross-lingual Word Embedding Models
• Compositional Morphology for Word Representations and Language Modelling
• Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunities
• Meta-Path Guided Embedding for Similarity Search in Large-Scale Heterogeneous Information Networks
• Learning Latent Vector Spaces for Product Search
• How to Train Your Energy-Based Models
• Representation Learning for Natural Language Processing
• Hierarchical Neural Language Models for Joint Representation of Streaming Documents and their Content
• Eating Healthier: Exploring Nutrition Information for Healthier Recipe Recommendation
• MultiVERSE: a multiplex and multiplex-heterogeneous network embedding approach
• Automatic Speech Recognition with Very Large Conversational Finnish and Estonian Vocabularies
• A Study of the Generalizability of Self-Supervised Representations
• Machine Learning Methods Economists Should Know About
• Efficient Neural Query Auto Completion
• Learning and Transferring IDs Representation in E-commerce
• Medical Synonym Extraction with Concept Space Models
• Pretrained Generalized Autoregressive Model with Adaptive Probabilistic Label Clusters for Extreme Multi-label Text Classification
• How to Train Your Energy-Based Model for Regression
• Minimum Stein Discrepancy Estimators
• Auto-Sizing Neural Networks: With Applications to n-gram Language Models
• Exponential Family Estimation via Adversarial Dynamics Embedding
• Few-shot Visual Reasoning with Meta-analogical Contrastive Learning
• Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency
• Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings
• Noise-Contrastive Estimation for Multivariate Point Processes
• Zoo Guide to Network Embedding
• An Empirical Study of Efficient ASR Rescoring with Transformers
• Vocabulary Manipulation for Neural Machine Translation
• Unsupervised and Efficient Vocabulary Expansion for Recurrent Neural Network Language Models in ASR
• Relaxed Softmax for learning from Positive and Unlabeled data
• Teaching Machines to Converse
• Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
• Ordering-sensitive and Semantic-aware Topic Modeling
• A Survey On Neural Word Embeddings
• Model Interpolation with Trans-dimensional Random Field Language Models for Speech Recognition
• Probabilistic Modelling of Morphologically Rich Languages
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
• Word Embedding Algorithms as Generalized Low Rank Models and their Canonical Form
• Efficient Contextual Representation Learning Without Softmax Layer
• Group-wise Contrastive Learning for Neural Dialogue Generation
• Neural Machine Translation: A Review and Survey
• Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling
• Listen, Attend, Spell and Adapt: Speaker Adapted Sequence-to-Sequence ASR
• Scalable Bayesian Modelling of Paired Symbols
• Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models
• Fine-tuning of Language Models with Discriminator
• Free Energy Node Embedding via Generalized Skip-gram with Negative Sampling
• Understanding Hard Negatives in Noise Contrastive Estimation
• Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
• Generative Max-Mahalanobis Classifiers for Image Classification, Generation and More
• Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation