InCoder: A Generative Model for Code Infilling and Synthesis

Abstract

Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. The InCoder models and code are publicly released. https://sites.google.com/view/incoder-code-models

ICLR 2023. v3: camera-ready that includes PLBART and OpenAI baselines

Cited by in corpus (21)

• Practical Program Repair in the Era of Large Pre-trained Language Models
• Regression Transformer: Concurrent sequence regression and generation for molecular language modeling
• Large Language Models for Code: Security Hardening and Adversarial Testing
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
• How Effective Are Neural Networks for Fixing Security Vulnerabilities
• LLM for SoC Security: A Paradigm Shift
• Synthetic Data Generation Using Large Language Models: Advances in Text and Code
• Enhanced Automated Code Vulnerability Repair using Large Language Models
• CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
• Revisiting the Plastic Surgery Hypothesis via Large Language Models
• ChatGPT Code Detection: Techniques for Uncovering the Source of Code
• Isolating Compiler Bugs by Generating Effective Witness Programs with Large Language Models
• Diverse Title Generation for Stack Overflow Posts with Multiple Sampling Enhanced Transformer
• PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
• CoEdPilot: Recommending Code Edits with Learned Prior Edit Relevance, Project-wise Awareness, and Interactive Nature
• DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models
• Insights into resource utilization of code small language models serving with runtime engines and execution providers
• The Effectiveness of Security Interventions on GitHub
• Deduplicating and Ranking Solution Programs for Suggesting Reference Solutions
• INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers