Zero-shot Text Classification With Generative Language Models

Abstract

This work investigates the use of natural language to enable zero-shot model adaptation to new tasks. We use text and metadata from social commenting platforms as a source for a simple pretraining task. We then provide the language model with natural language descriptions of classification tasks as input and train it to generate the correct answer in natural language via a language modeling objective. This allows the model to generalize to new classification tasks without the need for multiple multitask classification heads. We show the zero-shot performance of these generative language models, trained with weak supervision, on six benchmark text classification datasets from the torchtext library. Despite no access to training data, we achieve up to a 45% absolute improvement in classification accuracy over random or majority class baselines. These results show that natural language can serve as simple and powerful descriptors for task adaptation. We believe this points the way to new metalearning strategies for text problems.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• The Natural Language Decathlon: Multitask Learning as Question Answering
• One-shot Learning with Memory-Augmented Neural Networks
• Hierarchical Neural Story Generation
• Meta-Learning Update Rules for Unsupervised Representation Learning

Cited by in corpus (14)

• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Information Retrieval: Recent Advances and Beyond
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Entailment as Few-Shot Learner
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Document Ranking with a Pretrained Sequence-to-Sequence Model
• Framing the News:From Human Perception to Large Language Model Inferences
• Match-Prompt: Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning
• Automated Essay Scoring Using Transformer Models
• Contextual Lensing of Universal Sentence Representations
• Science Checker: Extractive-Boolean Question Answering For Scientific Fact Checking
• Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction
• Semi-Supervised Exaggeration Detection of Health Science Press Releases
• NatCat: Weakly Supervised Text Classification with Naturally Annotated Resources