Making Pre-trained Language Models Better Few-shot Learners

Abstract

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF--better few-shot fine-tuning of language models--a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.

arXiv commentAccepted to ACL 2021. The code is publicly available at https://github.com/princeton-nlp/LM-BFF

References in corpus (12)

• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Matching Networks for One Shot Learning
• A Simple Method for Commonsense Reasoning
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• Revisiting Few-sample BERT Fine-tuning
• Language Models as Knowledge Bases?
• Hyperspherical Prototype Networks
• Few-shot Text Classification with Distributional Signatures
• Commonsense Knowledge Mining from Pretrained Models
• Factual Probing Is [MASK]: Learning vs. Learning to Recall

Cited by in corpus (28)

• Learning Transferable Visual Models From Natural Language Supervision
• Learning to Prompt for Vision-Language Models
• Pre-trained Models for Natural Language Processing: A Survey
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• True Few-Shot Learning with Language Models
• Unifying Vision-and-Language Tasks via Text Generation
• Entailment as Few-Shot Learner
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5
• PTR: Prompt Tuning with Rules for Text Classification
• Factual Probing Is [MASK]: Learning vs. Learning to Recall
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• Template-Based Named Entity Recognition Using BART
• Primer: Searching for Efficient Transformers for Language Modeling
• Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
• How to Query Language Models?
• Enhancing Low-Resource Relation Representations through Multi-View Decoupling
• BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?
• Learning to Ask for Data-Efficient Event Argument Extraction
• Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction
• Few-Sample Named Entity Recognition for Security Vulnerability Reports by Fine-Tuning Pre-Trained Language Models
• Do Language Models Perform Generalizable Commonsense Inference?
• Solving Aspect Category Sentiment Analysis as a Text Generation Task
• A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters
• ConQX: Semantic Expansion of Spoken Queries for Intent Detection based on Conditioned Text Generation