No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence

Abstract

Pre-trained models have been shown effective in many code intelligence tasks. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream data, while in practice, the scenarios with scarce data are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this paper, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with three code intelligence tasks including defect prediction, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all three tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26\% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data.

arXiv commentAccepted in ESEC/FSE 2022

References in corpus (19)

• Decoupled Weight Decay Regularization
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• BARTScore: Evaluating Generated Text as Text Generation
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• Easy over Hard: A Case Study on Deep Learning
• Learning and Evaluating Contextual Embedding of Source Code
• PPT: Pre-trained Prompt Tuning for Few-shot Learning
• How Context Affects Language Models' Factual Predictions
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
• Retrieval-Augmented Generation for Code Summarization via Hybrid GNN
• OpenPrompt: An Open-source Framework for Prompt-learning
• Cross-Domain Deep Code Search with Meta Learning
• Noisy Channel Language Model Prompting for Few-Shot Text Classification
• CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computing
• ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking

Cited by in corpus (9)

• Fuzz4All: Universal Fuzzing with Large Language Models
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
• On the Effectiveness of Large Language Models in Domain-Specific Code Generation
• Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials
• RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair
• On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code
• Software Model Evolution with Large Language Models: Experiments on Simulated, Public, and Industrial Datasets
• Prompt engineering and its implications on the energy consumption of Large Language Models
• Contextualized Data-Wrangling Code Generation in Computational Notebooks