103 citations · 179 across the 8 of their papers we have counts for
11 papers
A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards
Andreas Plesner, Curtis Northcutt, Francisco Guzmán +1
When post-training large language models on tasks with semi-verifiable rewards, there are many factors (training steps, base model size, training order, data quality, verifier accu…
StudentBench: AI and human tutoring yield equivalent GRE learning gains
Curtis Northcutt, Inaara Hasmani, Kevin Feng +3
Artificial intelligence offers an unprecedented opportunity to augment human capabilities, yet progress at the frontier has focused primarily on advancing model capabilities. We in…
BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows
Elaine Lau, Markus Dücker, Ronak Chaudhary +24
Existing AI benchmarks lack the fidelity to assess economically meaningful progress on professional workflows. To evaluate frontier AI agents in a high-value, labor-intensive profe…
DMLR: Data-centric Machine Learning Research -- Past, Present and Future
Luis Oala, Manil Maskey, Lilith Bat-Leah +35
Drawing from discussions at the inaugural DMLR workshop at ICML 2023 and meetings prior, in this report we outline the relevance of community engagement and infrastructure developm…
Identifying Incorrect Annotations in Multi-Label Classification Data
Aditya Thyagarajan, Elías Snorrason, Curtis Northcutt +1
In multi-label classification, each example in a dataset may be annotated as belonging to one or more classes (or none of the classes). Example applications include image (or docum…
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
Curtis G. Northcutt, Anish Athalye, Jonas Mueller
We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these labe…
Rapformer: Conditional Rap Lyrics Generation with Denoising Autoencoders
Nikola I. Nikolov, Eric Malmi, Curtis G. Northcutt +1
The ability to combine symbols to generate language is a defining characteristic of human intelligence, particularly in the context of artistic story-telling through lyrics. We dev…
Comment Ranking Diversification in Forum Discussions
Curtis G. Northcutt, Kimberly A. Leon, Naichun Chen
Viewing consumption of discussion forums with hundreds or more comments depends on ranking because most users only view top-ranked comments. When comments are ranked by an ordered…
Confident Learning: Estimating Uncertainty in Dataset Labels
Curtis G. Northcutt, Lu Jiang, Isaac L. Chuang
Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) is an alternative approach which…
Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels
Curtis G. Northcutt, Tailin Wu, Isaac L. Chuang
Noisy PN learning is the problem of binary classification when training examples may be mislabeled (flipped) uniformly with noise rate rho1 for positive examples and rho0 for negat…
Detecting and Preventing "Multiple-Account" Cheating in Massive Open Online Courses
Curtis G. Northcutt, Andrew D. Ho, Isaac L. Chuang
We describe a cheating strategy enabled by the features of massive open online courses (MOOCs) and detectable by virtue of the sophisticated data systems that MOOCs provide. The st…