Mining Idioms from Source Code

Abstract

We present the first method for automatically mining code idioms from a corpus of previously written, idiomatic software projects. We take the view that a code idiom is a syntactic fragment that recurs across projects and has a single semantic role. Idioms may have metavariables, such as the body of a for loop. Modern IDEs commonly provide facilities for manually defining idioms and inserting them on demand, but this does not help programmers to write idiomatic code in languages or using libraries with which they are unfamiliar. We present HAGGIS, a system for mining code idioms that builds on recent advanced techniques from statistical natural language processing, namely, nonparametric Bayesian probabilistic tree substitution grammars. We apply HAGGIS to several of the most popular open source projects from GitHub. We present a wide range of evidence that the resulting idioms are semantically meaningful, demonstrating that they do indeed recur across software projects and that they occur more frequently in illustrative code examples collected from a Q&A site. Manual examination of the most common idioms indicate that they describe important program concepts, including object creation, exception handling, and resource management.

References in corpus (2)

• Learning Natural Coding Conventions
• Structured Generative Models of Natural Source Code

Cited by in corpus (32)

• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges
• A Survey on Large Language Models for Code Generation
• Structured Generative Models of Natural Source Code
• CODIT: Code Editing with Tree-Based Neural Models
• CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences
• Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks
• Learning Python Code Suggestion with a Sparse Pointer Network
• Learning to Generate Corrective Patches using Neural Machine Translation
• CodeGRU: Context-aware Deep Learning with Gated Recurrent Unit for Source Code Modeling
• Top-Down Synthesis for Library Learning
• Code Prediction by Feeding Trees to Transformers
• Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example
• Learning API Usages from Bytecode: A Statistical Approach
• Towards Demystifying Dimensions of Source Code Embeddings
• Neural Attribute Machines for Program Generation
• JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation
• Tailored Mutants Fit Bugs Better
• Toward Programming Languages for Reasoning: Humans, Symbolic Systems, and AI Agents
• Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models
• On Repair with Probabilistic Attribute Grammars
• On the "Naturalness" of Buggy Code
• Advanced Graph-Based Deep Learning for Probabilistic Type Inference
• PLAID: Supporting Computing Instructors to Identify Domain-Specific Programming Plans at Scale
• Do People Prefer "Natural" code?
• Data-Driven Evidence-Based Syntactic Sugar Design
• Borrowing from Similar Code: A Deep Learning NLP-Based Approach for Log Statement Automation
• Comprehensive Integration of API Usage Patterns
• From API to NLI: A New Interface for Library Reuse
• Are My Invariants Valid? A Learning Approach
• Neural Program Generation Modulo Static Analysis
• Replacements and Replaceables: Making the Case for Code Variants