Contrastive estimation reveals topic posterior information to linear models

Abstract

Contrastive learning is an approach to representation learning that utilizes naturally occurring similar and dissimilar pairs of data points to find useful embeddings of data. In the context of document classification under topic modeling assumptions, we prove that contrastive learning is capable of recovering a representation of documents that reveals their underlying topic posterior information to linear models. We apply this procedure in a semi-supervised setup and demonstrate empirically that linear classifiers with these representations perform well in document classification tasks with very few training examples.

References in corpus (4)

• Scikit-learn: Machine Learning in Python
• Distributed Representations of Words and Phrases and their Compositionality
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Overparameterized Neural Networks Implement Associative Memory

Cited by in corpus (13)

• On the Opportunities and Risks of Foundation Models
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• Self-supervised Learning is More Robust to Dataset Imbalance
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
• Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Contrastive learning, multi-view redundancy, and linear models
• Investigating the Role of Negatives in Contrastive Representation Learning
• Leveraging Time Irreversibility with Order-Contrastive Pre-training
• Contrastive learning of strong-mixing continuous-time stochastic processes