AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

Abstract

The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AutoPrompt, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AutoPrompt, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more sophisticated and capable, potentially a replacement for finetuning.

v2: Fixed error in Figure 2

References in corpus (8)

• Decoupled Weight Decay Regularization
• Language Models are Few-Shot Learners
• A Simple Method for Commonsense Reasoning
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
• Unsupervised Question Answering by Cloze Translation
• Language Models as Knowledge Bases?
• Why Do Masked Neural Language Models Still Need Common Sense Knowledge?

Cited by in corpus (11)

• BARTScore: Evaluating Generated Text as Text Generation
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• Pretrained Transformers as Universal Computation Engines
• A Practical Survey on Zero-shot Prompt Design for In-context Learning
• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5
• Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
• Response Generation with Context-Aware Prompt Learning
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• Accurate, yet inconsistent? Consistency Analysis on Language Understanding Models
• Probing Pre-Trained Language Models for Disease Knowledge