Publications (12)
Evaluating Large Language Models Trained on Code
Mark Chen, Jerry Tworek, Heewoo Jun +55
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex p…
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Christopher Berner +24
On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as lo…
FoundationDB Record Layer: A Multi-Tenant Structured Datastore
Christos Chrysafis, Ben Collins, Scott Dugas +10
The FoundationDB Record Layer is an open source library that provides a record-oriented data store with semantics similar to a relational database implemented on top of FoundationD…
Fast Algorithms for Convolutional Neural Networks
Andrew Lavin, Scott Gray
Deep convolutional neural networks take GPU days of compute time to train on large data sets. Pedestrian detection for self driving cars requires very low latency. Image recognitio…
Scaling Laws for Autoregressive Generative Modeling
Tom Henighan, Jared Kaplan, Mor Katz +16
We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal imagetext models, and mathemat…
Generating Long Sequences with Sparse Transformers
Rewon Child, Scott Gray, Alec Radford +1
Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the at…