P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Abstract

Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning \cite{li2021prefix,qin2021learning} optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.Our code and data are released at https://github.com/THUDM/P-tuning-v2.

Proceedings of the 60th Annual Meeting of the Association of Computational Linguistics, 2022

References in corpus (14)

• Language Models are Few-Shot Learners
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• SentiPrompt: Sentiment Knowledge Enhanced Prompt-Tuning for Aspect-Based Sentiment Analysis
• NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark
• PTR: Prompt Tuning with Rules for Text Classification
• Factual Probing Is [MASK]: Learning vs. Learning to Recall
• Language Models are Good Translators
• TopicRefine: Joint Topic Prediction and Dialogue Response Generation for Multi-turn End-to-End Dialogue System
• Better Feature Integration for Named Entity Recognition

Cited by in corpus (14)

• No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence
• Dynamic Prefix-Tuning for Generative Template-based Event Extraction
• Prompt Learning for News Recommendation
• Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehension
• Federated Few-Shot Learning for Mobile NLP
• A Reliable Knowledge Processing Framework for Combustion Science using Foundation Models
• Foundation Models and Transformers for Anomaly Detection: A Survey
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• FE-Adapter: Adapting Image-based Emotion Classifiers to Videos
• ContrastNER: Contrastive-based Prompt Tuning for Few-shot NER
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction
• Collocation2Text: Controllable Text Generation from Guide Phrases in Russian
• Parameter-Efficient Methods for Metastases Detection from Clinical Notes