Evaluating Large Language Models Trained on Code

Abstract

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

corrected typos, added references, added authors, added acknowledgements

References in corpus (16)

• Learning Transferable Visual Models From Natural Language Supervision
• Scaling Laws for Neural Language Models
• Zero-Shot Text-to-Image Generation
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Generating Long Sequences with Sparse Transformers
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• DeepCoder: Learning to Write Programs
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• Carbon Emissions and Large Neural Network Training
• Jukebox: A Generative Model for Music
• Unsupervised Translation of Programming Languages
• A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
• MERLOT: Multimodal Neural Script Knowledge Models
• Alignment of Language Agents
• Persistent Anti-Muslim Bias in Large Language Models
• SPoC: Search-based Pseudocode to Code

Cited by in corpus (10)

• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
• Cascaded Fast and Slow Models for Efficient Semantic Code Search
• Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs
• Goal-driven text descriptions for images
• Towards Universality in Multilingual Text Rewriting
• Automated Support for Unit Test Generation: A Tutorial Book Chapter
• Searching for Replacement Classes
• HYDRA -- Hyper Dependency Representation Attentions