164 citations · 307 across the 6 of their papers we have counts for
7 papers
TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts
Oliver Jaffe, Dane Sherburn
We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models. We define research taste as the ability to pick interesting problems to solve, de…
PaperBench: Evaluating AI's Ability to Replicate AI Research
Giulio Starace, Oliver Jaffe, Dane Sherburn +10
We introduce PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024 Spotlight and Oral papers fro…
GPT-4o System Card
OpenAI, :, Aaron Hurst +416
GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's…
Python workflow for segmenting multiphase flow in porous rocks
Catherine Spurin, Sharon Ellman, Dane Sherburn +2
X-ray micro-computed tomography (X-ray micro-CT) is widely employed to investigate flow phenomena in porous media, providing a powerful alternative to core-scale experiments for es…
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe +9
We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kag…
Can Language Models Explain Their Own Classification Behavior?
Dane Sherburn, Bilal Chughtai, Owain Evans
Large language models (LLMs) perform well at a myriad of tasks, but explaining the processes behind this performance is a challenge. This paper investigates whether LLMs can give f…
Relational Graph Attention Networks
Dan Busbridge, Dane Sherburn, Pietro Cavallo +1
We investigate Relational Graph Attention Networks, a class of models that extends non-relational graph attention mechanisms to incorporate relational information, opening up these…