CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

Abstract

Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions. The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16.1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER. We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN in multi-turn fashion significantly improves program synthesis over that provided as a single turn. We make the training library JAXFORMER and model checkpoints available as open source contribution: https://github.com/salesforce/CodeGen.

Cited by in corpus (23)

• Large Language Models for Code: Security Hardening and Adversarial Testing
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• How Effective Are Neural Networks for Fixing Security Vulnerabilities
• LLM for SoC Security: A Paradigm Shift
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
• (Security) Assertions by Large Language Models
• Fixing Hardware Security Bugs with Large Language Models
• The Impact of AI in Physics Education: A Comprehensive Review from GCSE to University Levels
• Synthetic Data Generation Using Large Language Models: Advances in Text and Code
• How Beginning Programmers and Code LLMs (Mis)read Each Other
• Fine-tuning and Utilization Methods of Domain-specific LLMs
• Vul-LMGNNs: Fusing language models and online-distilled graph neural networks for code vulnerability detection
• Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models
• Several categories of Large Language Models (LLMs): A Short Survey
• CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
• Revisiting the Plastic Surgery Hypothesis via Large Language Models
• A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
• Metamorphic Malware Evolution: The Potential and Peril of Large Language Models
• Deduplicating and Ranking Solution Programs for Suggesting Reference Solutions
• INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
• Iterative Self-Training for Code Generation via Reinforced Re-Ranking
• Program Skeletons for Automated Program Translation
• Towards Top-Down Automated Development in Limited Scopes: A Neuro-Symbolic Framework from Expressibles to Executables