9 papers
AutoEval Done Right: Using Synthetic Data for Model Evaluation
Pierre Boyeau, Anastasios N. Angelopoulos, Nir Yosef +2
The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming. AI-labeled synthetic data can be used to decrease the number of h…
Theoretical Foundations of Conformal Prediction
Anastasios N. Angelopoulos, Rina Foygel Barber, Stephen Bates
This book is about conformal prediction and related inferential techniques that build on permutation tests and exchangeability. These techniques are useful in a diverse array of ta…
Conformal Risk Control for Non-Monotonic Losses
Anastasios N. Angelopoulos
Conformal risk control is an extension of conformal prediction for controlling risk functions beyond miscoverage. The original algorithm controls the expected value of a loss that…
Conformal Risk Control
Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch +2
We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarante…
Cost-Optimal Active AI Model Evaluation
Anastasios N. Angelopoulos, Jacob Eisenstein, Jonathan Berant +2
The development lifecycle of generative AI systems requires continual evaluation, data acquisition, and annotation, which is costly in both resources and time. In practice, rapid i…
Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design
Clara Fannjiang, Stephen Bates, Anastasios N. Angelopoulos +2
Many applications of machine learning methods involve an iterative protocol in which data are collected, a model is trained, and then outputs of that model are used to choose what…