True Few-Shot Learning with Language Models

Abstract

Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly-selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.

Code at https://github.com/ethanjperez/true_few_shot

References in corpus (20)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Learning Transferable Visual Models From Natural Language Supervision
• Prototypical Networks for Few-shot Learning
• Language Models are Few-Shot Learners
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• DARTS: Differentiable Architecture Search
• A tutorial introduction to the minimum description length principle
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• Meta-Learning with Implicit Gradients
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Making Pre-trained Language Models Better Few-shot Learners
• Entailment as Few-Shot Learner
• Generative Data Augmentation for Commonsense Reasoning
• A Surprisingly Robust Trick for Winograd Schema Challenge
• Few-Shot Text Generation with Pattern-Exploiting Training
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize
• The Description Length of Deep Learning Models
• Factual Probing Is [MASK]: Learning vs. Learning to Recall

Cited by in corpus (19)

• On the Opportunities and Risks of Foundation Models
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• Generating Test Scenarios from NL Requirements using Retrieval-Augmented LLMs: An Industrial Study
• HTLM: Hyper-Text Pre-Training and Prompting of Language Models
• Stance Detection: A Practical Guide to Classifying Political Beliefs in Text
• AlpaPICO: Extraction of PICO Frames from Clinical Trial Documents Using LLMs
• Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection
• PromptORE -- A Novel Approach Towards Fully Unsupervised Relation Extraction
• RAFT: A Real-World Few-Shot Text Classification Benchmark
• TAPE: Assessing Few-shot Russian Language Understanding
• Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs
• Relational World Knowledge Representation in Contextual Language Models: A Review
• Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
• Analyzing Commonsense Emergence in Few-shot Knowledge Models