A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks

Abstract

Autoregressive language models, pretrained using large text corpora to do well on next word prediction, have been successful at solving many downstream tasks, even with zero-shot usage. However, there is little theoretical understanding of this success. This paper initiates a mathematical study of this phenomenon for the downstream task of text classification by considering the following questions: (1) What is the intuitive connection between the pretraining task of next word prediction and text classification? (2) How can we mathematically formalize this connection and quantify the benefit of language modeling? For (1), we hypothesize, and verify empirically, that classification tasks of interest can be reformulated as sentence completion tasks, thus making language modeling a meaningful pretraining task. With a mathematical formalization of this hypothesis, we make progress towards (2) and show that language models that are $\epsilon$-optimal in cross-entropy (log-perplexity) learn features that can linearly solve such classification tasks with $\mathcal{O}(\sqrt{\epsilon })$ error, thus demonstrating that doing well on language modeling can be beneficial for downstream tasks. We experimentally verify various assumptions and theoretical findings, and also use insights from the analysis to design a new objective function that performs well on some classification tasks.

arXiv commentThis version is the camera-ready version for ICLR 2021. Main changes include a detailed discussion about natural tasks, more detailed proof sketch and updated experimental evaluations

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Skip-Thought Vectors
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Pointer Sentinel Mixture Models
• An efficient framework for learning sentence representations
• Learning to Generate Reviews and Discovering Sentiment
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Inferring Networks of Substitutable and Complementary Products
• Zero-shot Text Classification With Generative Language Models
• Notes on Noise Contrastive Estimation and Negative Sampling
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Contrastive estimation reveals topic posterior information to linear models

Cited by in corpus (5)

• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
• On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
• AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data