Measuring Coding Challenge Competence With APPS

Abstract

While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements.

arXiv commentNeurIPS 2021. Code and the APPS dataset is available at https://github.com/hendrycks/apps

References in corpus (11)

• Decoupled Weight Decay Regularization
• Evaluating Large Language Models Trained on Code
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• Measuring Mathematical Problem Solving With the MATH Dataset
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• Making Neural Programming Architectures Generalize via Recursion
• Unsupervised Translation of Programming Languages
• Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
• Generative Language Modeling for Automated Theorem Proving
• HellaSwag: Can a Machine Really Finish Your Sentence?

Cited by in corpus (14)

• Evaluating Large Language Models Trained on Code
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
• Synthetic Data Generation Using Large Language Models: Advances in Text and Code
• CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
• Diverse Title Generation for Stack Overflow Posts with Multiple Sampling Enhanced Transformer
• Leveraging Automated Unit Tests for Unsupervised Code Translation
• PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
• Mokav: Execution-driven Differential Testing with LLMs
• Piloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?
• Flows: Building Blocks of Reasoning and Collaborating AI
• NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands
• Programming Puzzles
• UNICS: Multilingual Code Search via Unified Pseudocode and Contrastive Transfer Learning