A Mutual Information Maximization Perspective of Language Representation Learning

Abstract

We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence). Our formulation provides an alternative perspective that unifies classical word embedding models (e.g., Skip-gram) and modern contextual embeddings (e.g., BERT, XLNet). In addition to enhancing our theoretical understanding of these methods, our derivation leads to a principled framework that can be used to construct new self-supervised tasks. We provide an example by drawing inspirations from related methods based on mutual information maximization that have been successful in computer vision, and introduce a simple self-supervised objective that maximizes the mutual information between a global sentence representation and n-grams in the sentence. Our analysis offers a holistic view of representation learning methods to transfer knowledge and translate progress across multiple domains (e.g., natural language processing, computer vision, audio processing).

arXiv comment12 pages, 3 figures

References in corpus (10)

• Distributed Representations of Words and Phrases and their Compositionality
• Learning deep representations by mutual information estimation and maximization
• Cross-lingual Language Model Pretraining
• Learning Representations by Maximizing Mutual Information Across Views
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Semi-supervised Sequence Learning
• MINE: Mutual Information Neural Estimation
• On Mutual Information Maximization for Representation Learning
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Learning and Evaluating General Linguistic Intelligence

Cited by in corpus (32)

• Pre-trained Models for Natural Language Processing: A Survey
• Contrastive Representation Learning: A Framework and Review
• What Makes for Good Views for Contrastive Learning?
• An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Hyperspectral Image Classification With Contrastive Graph Convolutional Network
• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
• On Mutual Information in Contrastive Learning for Visual Representations
• The Effect of Natural Distribution Shift on Question Answering Models
• Coreferential Reasoning Learning for Language Representation
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• Fast-adapting and Privacy-preserving Federated Recommender System
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
• On Learning Universal Representations Across Languages
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• Neural Methods for Point-wise Dependency Estimation
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• Hybrid Generative-Contrastive Representation Learning
• Telescoping Density-Ratio Estimation
• The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
• Self-supervised Representation Learning with Relative Predictive Coding
• Evaluating Modules in Graph Contrastive Learning
• ProtoInfoMax: Prototypical Networks with Mutual Information Maximization for Out-of-Domain Detection
• Beyond Individual Input for Deep Anomaly Detection on Tabular Data
• On Sampling-Based Training Criteria for Neural Language Modeling
• Self-Supervised Graph Learning with Proximity-based Views and Channel Contrast
• Cross-modal Image Retrieval with Deep Mutual Information Maximization
• Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures