312 citations · 694 across the 17 of their papers we have counts for
6 papers · 1 filter
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
Elliot Glazer, Ege Erdil, Tamay Besiroglu +21
We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most…
Estimating Idea Production: A Methodological Survey
Ege Erdil, Tamay Besiroglu, Anson Ho
Accurately modeling the production of new ideas is crucial for innovation theory and endogenous growth models. This paper provides a comprehensive methodological survey of strategi…
Chinchilla Scaling: A replication attempt
Tamay Besiroglu, Ege Erdil, Matthew Barnett +1
Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parame…
Assessing Human Judgment Forecasts in the Rapid Spread of the Mpox Outbreak: Insights and Challenges for Pandemic Preparedness
Thomas McAndrew, Maimuna S. Majumder, Andrew A. Lover +10
In May 2022, mpox (formerly monkeypox) spread to non-endemic countries rapidly. Human judgment is a forecasting approach that has been sparsely evaluated during the beginning of an…
Algorithmic progress in language models
Anson Ho, Tamay Besiroglu, Ege Erdil +6
We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations…
The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?
Tamay Besiroglu, Sage Andrus Bergerson, Amelia Michael +3
There are pronounced differences in the extent to which industrial and academic AI labs use computing resources. We provide a data-driven survey of the role of the compute divide i…