Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Abstract

This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning". Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x' that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string x, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: it allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this paper we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g.the choice of pre-trained models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts, but also release other resources, e.g., a website http://pretrain.nlpedia.ai/ including constantly-updated survey, and paperlist.

Website: http://pretrain.nlpedia.ai/

Cited by in corpus (37)

• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding
• No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence
• RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
• An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
• Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting
• Prompt Learning for News Recommendation
• Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
• Augmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting
• Mask-guided BERT for Few Shot Text Classification
• Understanding EFL Student Idea Generation Strategies for Creative Writing with NLG Tools
• CLAMP: Prompt-based Contrastive Learning for Connecting Language and Animal Pose
• Few-Shot Stance Detection via Target-Aware Prompt Distillation
• ScatterShot: Interactive In-context Example Curation for Text Transformation
• Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?
• Framing the News:From Human Perception to Large Language Model Inferences
• Match-Prompt: Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning
• P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
• Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives
• MetaTroll: Few-shot Detection of State-Sponsored Trolls with Transformer Adapters
• ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning
• AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• GPT Struct Me: Probing GPT Models on Narrative Entity Extraction
• PromptORE -- A Novel Approach Towards Fully Unsupervised Relation Extraction
• Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention
• A Novel Prompt-tuning Method: Incorporating Scenario-specific Concepts into a Verbalizer
• Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction
• IDIAPers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach
• OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models
• Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation
• TAPE: Assessing Few-shot Russian Language Understanding
• Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics
• Leveraging Label Correlations in a Multi-label Setting: A Case Study in Emotion
• PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models
• Customizing ChatGPT for Second Language Speaking Practice: Genuine Support or Just a Marketing Gimmick?
• LLM-GROP: Visually Grounded Robot Task and Motion Planning with Large Language Models