Publications (12)
A Rosetta Stone for AI Benchmarks
Anson Ho, Jean-Stanislas Denain, David Atanasov +2
Most AI benchmarks saturate within years or even months after they are introduced, making it hard to study long-run trends in AI capabilities. To address this challenge, we build a…
International AI Safety Report
Yoshua Bengio, Sören Mindermann, Daniel Privitera +93
The first International AI Safety Report comprehensively synthesizes the current evidence on the capabilities, risks, and safety of advanced AI systems. The report was mandated by…
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
Elliot Glazer, Ege Erdil, Tamay Besiroglu +21
We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most…
Estimating Idea Production: A Methodological Survey
Ege Erdil, Tamay Besiroglu, Anson Ho
Accurately modeling the production of new ideas is crucial for innovation theory and endogenous growth models. This paper provides a comprehensive methodological survey of strategi…
Compute Trends Across Three Eras of Machine Learning
Jaime Sevilla, Lennart Heim, Anson Ho +3
Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily…
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Tilman Räuker, Anson Ho, Stephen Casper +1
The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, th…