Adversarial Contrastive Estimation
arXiv:1805.03642
Abstract
Learning by contrasting positive and negative samples is a general strategy adopted by many methods. Noise contrastive estimation (NCE) for word embeddings and translating embeddings for knowledge graphs are examples in NLP employing this approach. In this work, we view contrastive learning as an abstraction of all such methods and augment the negative sampler into a mixture distribution containing an adversarially learned sampler. The resulting adaptive sampler finds harder negative examples, which forces the main model to learn a better representation of the data. We evaluate our proposal on learning word embeddings, order embeddings and knowledge graph embeddings and observe both faster convergence and improved results on multiple metrics.
arXiv commentAssociation for Computational Linguistics, 2018
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Categorical Reparameterization with Gumbel-Softmax
- Improved Training of Wasserstein GANs
- Complex Embeddings for Simple Link Prediction
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Energy-based Generative Adversarial Network
- Efficient softmax approximation for GPUs
- Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
- Contrastive Learning for Image Captioning
- On distinguishability criteria for estimating generative models
- Notes on Noise Contrastive Estimation and Negative Sampling
- Learning Approximate Inference Networks for Structured Prediction
- KBGAN: Adversarial Learning for Knowledge Graph Embeddings
- Augment and Reduce: Stochastic Inference for Large Categorical Distributions
Cited by in corpus (9)
- Semantic Models for the First-stage Retrieval: A Comprehensive Review
- Contrastive Learning with Adversarial Examples
- Understanding Negative Sampling in Graph Representation Learning
- Viewmaker Networks: Learning Views for Unsupervised Representation Learning
- Energy-Based Imitation Learning
- Flow Contrastive Estimation of Energy-Based Models
- Contrastive Attraction and Contrastive Repulsion for Representation Learning
- Adversarial Contrastive Pre-training for Protein Sequences
- Identifiable Energy-based Representations: An Application to Estimating Heterogeneous Causal Effects