Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models

Abstract

Recent advancements in source code summarization have leveraged transformer-based pre-trained models, including Large Language Models of Code (LLMCs), to automate and improve the generation of code summaries. However, existing methods often focus on generating a single high-quality summary for a given source code, neglecting scenarios where the generated summary might be inadequate and alternative options are needed. In this paper, we introduce Variational Prefix Tuning (VPT), a novel approach that enhances pre-trained models' ability to generate diverse yet accurate sets of summaries, allowing the user to choose the most suitable one for the given source code. Our method integrates a Conditional Variational Autoencoder (CVAE) framework as a modular component into pre-trained models, enabling us to model the distribution of observed target summaries and sample continuous embeddings to be used as prefixes to steer the generation of diverse outputs during decoding. Importantly, we construct our method in a parameter-efficient manner, eliminating the need for expensive model retraining, especially when using LLMCs. Furthermore, we employ a bi-criteria reranking method to select a subset of generated summaries, optimizing both the diversity and the accuracy of the options presented to users. We present extensive experimental evaluations using widely used datasets and current state-of-the-art pre-trained code summarization models to demonstrate the effectiveness of our approach and its adaptability across models.

Accepted by the Journal of Systems and Software

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Generative Adversarial Networks
• Evaluating Large Language Models Trained on Code
• The Curious Case of Neural Text Degeneration
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Deep Recurrent Quantization for Generating Sequential Binary Codes
• Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval
• Code Llama: Open Foundation Models for Code
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Texygen: A Benchmarking Platform for Text Generation Models
• Code Structure Guided Transformer for Source Code Summarization
• A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• Variational Transformers for Diverse Response Generation
• Transformer-based Conditional Variational Autoencoder for Controllable Story Generation