312 citations · 445 across the 7 of their papers we have counts for
7 papers
GATE: An Integrated Assessment Model for AI Automation
Ege Erdil, Andrei Potlogea, Tamay Besiroglu +6
Assessing the economic impacts of artificial intelligence requires integrating insights from both computer science and economics. We present the Growth and AI Transition Endogenous…
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
Elliot Glazer, Ege Erdil, Tamay Besiroglu +21
We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most…
Algorithmic progress in language models
Anson Ho, Tamay Besiroglu, Ege Erdil +6
We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations…
Power Law Trends in Speedrunning and Machine Learning
Ege Erdil, Jaime Sevilla
We find that improvements in speedrunning world records follow a power law pattern. Using this observation, we answer an outstanding question from previous work: How do we improve…
Will we run out of data? Limits of LLM scaling based on human-generated data
Pablo Villalobos, Anson Ho, Jaime Sevilla +3
We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on cur…
Machine Learning Model Sizes and the Parameter Gap
Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu +3
We study trends in model size of notable machine learning systems over time using a curated dataset. From 1950 to 2018, model size in language models increased steadily by seven or…
Compute Trends Across Three Eras of Machine Learning
Jaime Sevilla, Lennart Heim, Anson Ho +3
Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily…