Diet Code Is Healthy: Simplifying Programs for Pre-trained Models of Code

Abstract

Pre-trained code representation models such as CodeBERT have demonstrated superior performance in a variety of software engineering tasks, yet they are often heavy in complexity, quadratically with the length of the input sequence. Our empirical analysis of CodeBERT's attention reveals that CodeBERT pays more attention to certain types of tokens and statements such as keywords and data-relevant statements. Based on these findings, we propose DietCode, which aims at lightweight leverage of large pre-trained models for source code. DietCode simplifies the input program of CodeBERT with three strategies, namely, word dropout, frequency filtering, and an attention-based strategy which selects statements and tokens that receive the most attention weights during pre-training. Hence, it gives a substantial reduction in the computational cost without hampering the model performance. Experimental results on two downstream tasks show that DietCodeBERT provides comparable results to CodeBERT with 40% less computational cost in fine-tuning and testing.

Accepted to be published in ESEC/FSE 2022

References in corpus (6)

• Cross-lingual Language Model Pretraining
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• Query Expansion Based on Crowd Knowledge for Code Search
• Understanding Neural Code Intelligence Through Program Simplification
• Learned Token Pruning for Transformers
• What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code

Cited by in corpus (4)

• Greening Large Language Models of Code
• Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarization
• ALPINE: An adaptive language-agnostic pruning method for language models for code
• Study of Distractors in Neural Models of Code