Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation

Abstract

Noise-contrastive estimation (NCE) is a statistically consistent method for learning unnormalized probabilistic models. It has been empirically observed that the choice of the noise distribution is crucial for NCE's performance. However, such observations have never been made formal or quantitative. In fact, it is not even clear whether the difficulties arising from a poorly chosen noise distribution are statistical or algorithmic in nature. In this work, we formally pinpoint reasons for NCE's poor performance when an inappropriate noise distribution is used. Namely, we prove these challenges arise due to an ill-behaved (more precisely, flat) loss landscape. To address this, we introduce a variant of NCE called "eNCE" which uses an exponential loss and for which normalized gradient descent addresses the landscape issues provably when the target and noise distributions are in a given exponential family.

References in corpus (12)

• A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
• Implicit Generation and Generalization in Energy-Based Models
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Notes on Noise Contrastive Estimation and Negative Sampling
• Beyond Convexity: Stochastic Quasi-Convex Optimization
• Logistic Regression: Tight Bounds for Stochastic and Online Optimization
• Training Deep Energy-Based Models with f-Divergence Minimization
• Simple and optimal high-probability bounds for strongly-convex stochastic gradient descent
• On the accuracy of self-normalized log-linear models
• Telescoping Density-Ratio Estimation
• Generative Ratio Matching Networks
• Heavy-tailed Streaming Statistical Estimation