Naturalness of Attention: Revisiting Attention in Code Language Models

Mootez Saad, Tushar Sharma

Language models for code such as CodeBERT offer the capability to learn advanced source code representation, but their opacity poses barriers to understanding of captured propertie…