7 papers
Measuring Coding Challenge Competence With APPS
Dan Hendrycks, Steven Basart, Saurav Kadavath +8
While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importan…
Limitations of Post-Hoc Feature Alignment for Robustness
Collin Burns, Jacob Steinhardt
Feature alignment is an approach to improving robustness to distribution shift that matches the distribution of feature activations between the training distribution and test distr…
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
Dan Hendrycks, Collin Burns, Anya Chen +1
Many specialized domains remain untouched by deep learning, as large labeled datasets require expensive expert annotators. We address this bottleneck within the legal domain by int…
Measuring Mathematical Problem Solving With the MATH Dataset
Dan Hendrycks, Collin Burns, Saurav Kadavath +5
Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, w…
Measuring Massive Multitask Language Understanding
Dan Hendrycks, Collin Burns, Steven Basart +4
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attai…
Streaming Complexity of SVMs
Alexandr Andoni, Collin Burns, Yi Li +2
We study the space complexity of solving the bias-regularized SVM problem in the streaming model. This is a classic supervised learning problem that has drawn lots of attention, in…