Evaluating Large Language Models Trained on Code
arXiv:2107.03374
Abstract
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
arXiv commentcorrected typos, added references, added authors, added acknowledgements
References in corpus (34)
- Distributed Representations of Words and Phrases and their Compositionality
- Learning Transferable Visual Models From Natural Language Supervision
- Pixel Recurrent Neural Networks
- Scaling Laws for Neural Language Models
- Zero-Shot Text-to-Image Generation
- BEiT: BERT Pre-Training of Image Transformers
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Generating Long Sequences with Sparse Transformers
- Semi-supervised Sequence Learning
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
- Learning to Execute
- DeepCoder: Learning to Write Programs
- Neural GPUs Learn Algorithms
- CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
- Measuring Coding Challenge Competence With APPS
- Structured Generative Models of Natural Source Code
- Carbon Emissions and Large Neural Network Training
- Contrastive Code Representation Learning
- Jukebox: A Generative Model for Music
- Latent Predictor Networks for Code Generation
- Women's Participation in Open Source Software: A Survey of the Literature
- Calibrate Before Use: Improving Few-Shot Performance of Language Models
- Unsupervised Translation of Programming Languages
- A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
- Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
- MERLOT: Multimodal Neural Script Knowledge Models
- Alignment of Language Agents
- Learning to summarize from human feedback
- Learning Compositional Neural Programs with Recursive Tree Search and Planning
- Persistent Anti-Muslim Bias in Large Language Models
- SPoC: Search-based Pseudocode to Code
- Unit Test Case Generation with Transformers and Focal Context
Cited by in corpus (149)
- On the Opportunities and Risks of Foundation Models
- Generative AI
- Automatic Generation of Programming Exercises and Code Explanations using Large Language Models
- Practical Program Repair in the Era of Large Pre-trained Language Models
- Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
- SELFIES and the future of molecular string representations
- The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development
- ChatGPT and Software Testing Education: Promises & Perils
- Chip-Chat: Challenges and Opportunities in Conversational Hardware Design
- A Taxonomy of Prompt Modifiers for Text-To-Image Generation
- A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level
- Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
- Measuring Coding Challenge Competence With APPS
- Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
- Large Language Models for Code: Security Hardening and Adversarial Testing
- Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
- Machine Culture
- NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing
- "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
- Out of the BLEU: how should we assess quality of the Code Generation models?
- LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
- Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?
- Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?
- How Effective Are Neural Networks for Fixing Security Vulnerabilities
- ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
- PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
- LLM for SoC Security: A Paradigm Shift
- Unsolved Problems in ML Safety
- What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
- Tool Learning with Large Language Models: A Survey
- (Security) Assertions by Large Language Models
- Fixing Hardware Security Bugs with Large Language Models
- The Impact of AI in Physics Education: A Comprehensive Review from GCSE to University Levels
- ChatGPT in the classroom. Exploring its potential and limitations in a Functional Programming course
- Better Together? An Evaluation of AI-Supported Code Translation
- Finetuned Language Models Are Zero-Shot Learners
- LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
- Refactoring Programs Using Large Language Models with Few-Shot Examples
- LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systems
- Natural Language Processing Models That Automate Programming Will Transform Chemistry Research and Teaching
- Learning from flowsheets: A generative transformer model for autocompletion of flowsheets
- AI-driven Development Is Here: Should You Worry?
- Synthetic Data Generation Using Large Language Models: Advances in Text and Code
- ChatGPT and Other Large Language Models as Evolutionary Engines for Online Interactive Collaborative Game Design
- Language Models as Zero-Shot Trajectory Generators
- HPC-GPT: Integrating Large Language Model for High-Performance Computing
- ThoughtSource: A central hub for large language model reasoning data
- Many bioinformatics programming tasks can be automated with ChatGPT
- HPC-Coder: Modeling Parallel Programs using Large Language Models
- A Survey on Symbolic Knowledge Distillation of Large Language Models
- Fully Autonomous Programming with Large Language Models
- Automatically Generating CS Learning Materials with Large Language Models
- On the Effectiveness of Transfer Learning for Code Search
- A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Models
- CoverUp: Effective High Coverage Test Generation for Python
- GeoCode-GPT: A Large Language Model for Geospatial Code Generation Tasks
- A Unified Industrial Large Knowledge Model Framework in Industry 4.0 and Smart Manufacturing
- Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models
- Deploying and Evaluating LLMs to Program Service Mobile Robots
- Fuzzing Automatic Differentiation in Deep-Learning Libraries
- How People Prompt to Create Interactive VR Scenes
- Several categories of Large Language Models (LLMs): A Short Survey
- Automating the Correctness Assessment of AI-generated Code for Security Contexts
- PADL: Language-Directed Physics-Based Character Control
- Beyond Code Generation: LLM-supported Exploration of the Program Design Space
- Higher education assessment practice in the era of generative AI tools
- How Do Programming Students Use Generative AI?
- CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
- Revisiting the Plastic Surgery Hypothesis via Large Language Models
- Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models
- Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators
- Drastic Circuit Depth Reductions with Preserved Adversarial Robustness by Approximate Encoding for Quantum Machine Learning
- A Framework for Neurosymbolic Robot Action Planning using Large Language Models
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
- AI Challenges for Society and Ethics
- Survey of Code Search Based on Deep Learning
- A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
- Reliable Natural Language Understanding with Large Language Models and Answer Set Programming
- Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Study
- M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
- Streamlining evidence based clinical recommendations with large language models
- A User-centered Security Evaluation of Copilot
- The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification
- Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming
- Public Attitudes Toward ChatGPT on Twitter: Sentiments, Topics, and Occupations
- Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments
- Inductive Biases and Variable Creation in Self-Attention Mechanisms
- Automating Traffic Model Enhancement with AI Research Agent
- Toward a Theory of Causation for Interpreting Neural Code Models
- Large Language Models for Depression Recognition in Spoken Language Integrating Psychological Knowledge
- Metamorphic Malware Evolution: The Potential and Peril of Large Language Models
- Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT
- Diverse Title Generation for Stack Overflow Posts with Multiple Sampling Enhanced Transformer
- Exploring and Evaluating Personalized Models for Code Generation
- Programming-by-Demonstration for Long-Horizon Robot Tasks
- PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
- Learning to Synthesize Programs as Interpretable and Generalizable Policies
- User-Driven Support for Visualization Prototyping in D3
- Leveraging Automated Unit Tests for Unsupervised Code Translation
- Solving Novel Program Synthesis Problems with Genetic Programming using Parametric Polymorphism
- SPROUT: an Interactive Authoring Tool for Generating Programming Tutorials with the Visualization of Large Language Models
- Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models
- Normative Conflicts and Shallow AI Alignment
- Program Repair with Minimal Edits Using CodeT5
- GPT Struct Me: Probing GPT Models on Narrative Entity Extraction
- Scaffolding Progress: How Structured Editors Shape Novice Errors When Transitioning from Blocks to Text
- LASSI: An LLM-based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codes
- Mokav: Execution-driven Differential Testing with LLMs
- RepairBench: Leaderboard of Frontier Models for Program Repair
- Mirror: A Natural Language Interface for Data Querying, Summarization, and Visualization
- API-Assisted Code Generation for Question Answering on Varied Table Structures
- Cascaded Fast and Slow Models for Efficient Semantic Code Search
- Piloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?
- Lyra: A Benchmark for Turducken-Style Code Generation
- Enabling Automatic Repair of Source Code Vulnerabilities Using Data-Driven Methods
- Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models
- Complex QA and language models hybrid architectures, Survey
- Do Comments and Expertise Still Matter? An Experiment on Programmers' Adoption of AI-Generated JavaScript Code
- DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models
- Insights into resource utilization of code small language models serving with runtime engines and execution providers
- Knowledge Graph Based Repository-Level Code Generation
- Solving Probability and Statistics Problems by Program Synthesis
- Flows: Building Blocks of Reasoning and Collaborating AI
- BibSonomy Meets ChatLLMs for Publication Management: From Chat to Publication Management: Organizing your related work using BibSonomy & LLMs
- LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving
- Exploring the Effectiveness of Abstract Syntax Tree Patterns for Algorithm Recognition
- Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models
- Camera Control at the Edge with Language Models for Scene Understanding
- Analyzing Semantic Faithfulness of Language Models via Input Intervention on Question Answering
- Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming
- Towards Universality in Multilingual Text Rewriting
- Can Pre-trained Language Models be Used to Resolve Textual and Semantic Merge Conflicts?
- Co-Disclosing the Computer: LLM-Mediated Computing through Reflective Conversation
- Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs
- LLM-GROP: Visually Grounded Robot Task and Motion Planning with Large Language Models
- INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
- Solving Linear Algebra by Program Synthesis
- Goal-driven text descriptions for images
- CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding
- Automated Support for Unit Test Generation: A Tutorial Book Chapter
- HYDRA -- Hyper Dependency Representation Attentions
- Neural Program Generation Modulo Static Analysis
- On Inductive Biases for Machine Learning in Data Constrained Settings
- Capturing Structural Locality in Non-parametric Language Models
- Searching Personal Collections
- Towards A Measure Of General Machine Intelligence
- Learning to Superoptimize Real-world Programs
- Searching for Replacement Classes
- Linguacodus: A Synergistic Framework for Transformative Code Generation in Machine Learning Pipelines