Notes on Noise Contrastive Estimation and Negative Sampling

Abstract

Estimating the parameters of probabilistic models of language such as maxent models and probabilistic neural models is computationally difficult since it involves evaluating partition functions by summing over an entire vocabulary, which may be millions of word types in size. Two closely related strategies---noise contrastive estimation (Mnih and Teh, 2012; Mnih and Kavukcuoglu, 2013; Vaswani et al., 2013) and negative sampling (Mikolov et al., 2012; Goldberg and Levy, 2014)---have emerged as popular solutions to this computational problem, but some confusion remains as to which is more appropriate and when. This document explicates their relationships to each other and to other estimation techniques. The analysis shows that, although they are superficially similar, NCE is a general parameter estimation technique that is asymptotically unbiased, while negative sampling is best understood as a family of binary classification models that are useful for learning word representations but not as a general-purpose estimator.

4 pages

References in corpus (2)

• Natural Language Processing (almost) from Scratch
• word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method

Cited by in corpus (19)

• Compositional Fairness Constraints for Graph Embeddings
• Characterization of citizens using word2vec and latent topic analysis in a large set of tweets
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Adversarial Contrastive Estimation
• Combining Representation Learning with Tensor Factorization for Risk Factor Analysis - an application to Epilepsy and Alzheimer's disease
• Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding
• Long-span language modeling for speech recognition
• Calibrating and Improving Graph Contrastive Learning
• Improving Textual Network Embedding with Global Attention via Optimal Transport
• Sequential Adversarial Anomaly Detection for One-Class Event Data
• Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection In Task Oriented Dialog
• Neural Machine Translation: A Review and Survey
• PMI Matrix Approximations with Applications to Neural Language Modeling
• Evaluating a Generative Adversarial Framework for Information Retrieval
• GANE: A Generative Adversarial Network Embedding
• Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP
• Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation
• Weather Analogs with a Machine Learning Similarity Metric for Renewable Resource Forecasting
• On Sampling-Based Training Criteria for Neural Language Modeling