The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification

Abstract

The use of modern Natural Language Processing (NLP) techniques has shown to be beneficial for software engineering tasks, such as vulnerability detection and type inference. However, training deep NLP models requires significant computational resources. This paper explores techniques that aim at achieving the best usage of resources and available information in these models. We propose a generic approach, EarlyBIRD, to build composite representations of code from the early layers of a pre-trained transformer model. We empirically investigate the viability of this approach on the CodeBERT model by comparing the performance of 12 strategies for creating composite representations with the standard practice of only using the last encoder layer. Our evaluation on four datasets shows that several early layer combinations yield better performance on defect detection, and some combinations improve multi-class classification. More specifically, we obtain a +2 average improvement of detection accuracy on Devign with only 3 out of 12 layers of CodeBERT and a 3.3x speed-up of fine-tuning. These findings show that early layers can be used to obtain better results using the same resources, as well as to reduce resource usage during fine-tuning and inference.

arXiv commentThe content in this pre-print is the same as in the CRC accepted for publication in the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2023)

References in corpus (13)

• Evaluating Large Language Models Trained on Code
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
• Reducing Transformer Depth on Demand with Structured Dropout
• CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software
• StarCoder: may the source be with you!
• Neural Program Repair with Execution-based Backpropagation
• Learning and Evaluating Contextual Embedding of Source Code
• How to Fine-Tune BERT for Text Classification?
• On the Effect of Dropping Layers of Pre-trained Transformer Models
• Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing
• Break-It-Fix-It: Unsupervised Learning for Program Repair
• Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code

Cited by in corpus (1)

• Towards Integrating Emerging AI Applications in SE Education