What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?

Abstract

Pre-trained models of source code have gained widespread popularity in many code intelligence tasks. Recently, with the scaling of the model and corpus size, large language models have shown the ability of in-context learning (ICL). ICL employs task instructions and a few examples as demonstrations, and then inputs the demonstrations to the language models for making predictions. This new learning paradigm is training-free and has shown impressive performance in various natural language processing and code intelligence tasks. However, the performance of ICL heavily relies on the quality of demonstrations, e.g., the selected examples. It is important to systematically investigate how to construct a good demonstration for code-related tasks. In this paper, we empirically explore the impact of three key factors on the performance of ICL in code intelligence tasks: the selection, order, and number of demonstration examples. We conduct extensive experiments on three code intelligence tasks including code summarization, bug fixing, and program synthesis. Our experimental results demonstrate that all the above three factors dramatically impact the performance of ICL in code intelligence tasks. Additionally, we summarize our findings and provide takeaway suggestions on how to construct effective demonstrations, taking into account these three perspectives. We also show that a carefully-designed demonstration based on our findings can lead to substantial improvements over widely-used demonstration construction methods, e.g., improving BLEU-4, EM, and EM by at least 9.90%, 175.96%, and 50.81% on code summarization, bug fixing, and program synthesis, respectively

This paper is accepted by ASE 2023

References in corpus (17)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Evaluating Large Language Models Trained on Code
• Large Language Models are Zero-Shot Reasoners
• Emergent Abilities of Large Language Models
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• Practical Program Repair in the Era of Large Pre-trained Language Models
• PaLM-E: An Embodied Multimodal Language Model
• A Survey on In-context Learning
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence
• Synchromesh: Reliable code generation from pre-trained language models
• Selective Annotation Makes Language Models Better Few-Shot Learners
• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
• Binding Language Models in Symbolic Languages
• Recurrent Memory Transformer

Cited by in corpus (3)

• VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs
• Statically Contextualizing Large Language Models with Typed Holes
• The Few-shot Dilemma: Over-prompting Large Language Models