312 citations · 694 across the 17 of their papers we have counts for
18 papers
GATE: An Integrated Assessment Model for AI Automation
Ege Erdil, Andrei Potlogea, Tamay Besiroglu +6
Assessing the economic impacts of artificial intelligence requires integrating insights from both computer science and economics. We present the Growth and AI Transition Endogenous…
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
Elliot Glazer, Ege Erdil, Tamay Besiroglu +21
We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most…
Estimating Idea Production: A Methodological Survey
Ege Erdil, Tamay Besiroglu, Anson Ho
Accurately modeling the production of new ideas is crucial for innovation theory and endogenous growth models. This paper provides a comprehensive methodological survey of strategi…
Chinchilla Scaling: A replication attempt
Tamay Besiroglu, Ege Erdil, Matthew Barnett +1
Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parame…
Assessing Human Judgment Forecasts in the Rapid Spread of the Mpox Outbreak: Insights and Challenges for Pandemic Preparedness
Thomas McAndrew, Maimuna S. Majumder, Andrew A. Lover +10
In May 2022, mpox (formerly monkeypox) spread to non-endemic countries rapidly. Human judgment is a forecasting approach that has been sparsely evaluated during the beginning of an…
Algorithmic progress in language models
Anson Ho, Tamay Besiroglu, Ege Erdil +6
We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations…
The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?
Tamay Besiroglu, Sage Andrus Bergerson, Amelia Michael +3
There are pronounced differences in the extent to which industrial and academic AI labs use computing resources. We provide a data-driven survey of the role of the compute divide i…
Limits to the Energy Efficiency of CMOS Microprocessors
Anson Ho, Ege Erdil, Tamay Besiroglu
CMOS microprocessors have achieved massive energy efficiency gains but may reach limits soon. This paper presents an approach to estimating the limits on the maximum floating point…
Who is leading in AI? An analysis of industry AI research
Ben Cottier, Tamay Besiroglu, David Owen
AI research is increasingly industry-driven, making it crucial to understand company contributions to this field. We compare leading AI companies by research publications, citation…
Explosive growth from AI automation: A review of the arguments
Ege Erdil, Tamay Besiroglu
We examine whether substantial AI automation could accelerate global economic growth by about an order of magnitude, akin to the economic growth effects of the Industrial Revolutio…
Economic impacts of AI-augmented R&D
Tamay Besiroglu, Nicholas Emery-Xu, Neil Thompson
Since its emergence around 2010, deep learning has rapidly become the most important technique in Artificial Intelligence (AI), producing an array of scientific firsts in areas as…
Algorithmic progress in computer vision
Ege Erdil, Tamay Besiroglu
We investigate algorithmic progress in image classification on ImageNet, perhaps the most well-known test bed for computer vision. We estimate a model, informed by work on neural s…
Will we run out of data? Limits of LLM scaling based on human-generated data
Pablo Villalobos, Anson Ho, Jaime Sevilla +3
We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on cur…
Machine Learning Model Sizes and the Parameter Gap
Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu +3
We study trends in model size of notable machine learning systems over time using a curated dataset. From 1950 to 2018, model size in language models increased steadily by seven or…
Aggregating human judgment probabilistic predictions of COVID-19 transmission, burden, and preventative measures
Allison Codi, Damon Luk, David Braun +6
Aggregated human judgment forecasts for COVID-19 targets of public health importance are accurate, often outperforming computational models. Our work shows aggregated human judgmen…
Compute Trends Across Three Eras of Machine Learning
Jaime Sevilla, Lennart Heim, Anson Ho +3
Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily…
Chimeric forecasting: combining probabilistic predictions from computational models and human judgment
Thomas McAndrew, Allison Codi, Juan Cambeiro +5
Forecasts of the trajectory of an infectious agent can help guide public health decision making. A traditional approach to forecasting fits a computational model to structured data…
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Miles Brundage, Shahar Avin, Jasmine Wang +56
With the recent wave of progress in artificial intelligence (AI) has come a growing awareness of the large-scale impacts of AI systems, and recognition that existing regulations an…