75 citations · 347 across the 29 of their papers we have counts for
32 papers
Conformal Risk Control for Non-Monotonic Losses
Anastasios N. Angelopoulos
Conformal risk control is an extension of conformal prediction for controlling risk functions beyond miscoverage. The original algorithm controls the expected value of a loss that…
Cost-Optimal Active AI Model Evaluation
Anastasios N. Angelopoulos, Jacob Eisenstein, Jonathan Berant +2
The development lifecycle of generative AI systems requires continual evaluation, data acquisition, and annotation, which is costly in both resources and time. In practice, rapid i…
Gradient Equilibrium in Online Learning: Theory and Applications
Anastasios N. Angelopoulos, Michael I. Jordan, Ryan J. Tibshirani
We present a new perspective on online learning that we refer to as gradient equilibrium: a sequence of iterates achieves gradient equilibrium if the average of gradients of losses…
Theoretical Foundations of Conformal Prediction
Anastasios N. Angelopoulos, Rina Foygel Barber, Stephen Bates
This book is about conformal prediction and related inferential techniques that build on permutation tests and exchangeability. These techniques are useful in a diverse array of ta…
How to Evaluate Reward Models for RLHF
Evan Frick, Tianle Li, Connor Chen +6
We introduce a new benchmark for reward models that quantifies their ability to produce strong language models through RLHF (Reinforcement Learning from Human Feedback). The gold-s…
Automatically Adaptive Conformal Risk Control
Vincent Blot, Anastasios N Angelopoulos, Michael I Jordan +1
Science and technology have a growing need for effective mechanisms that ensure reliable, controlled performance from black-box machine learning algorithms. These performance guara…
Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction
Drew T. Nguyen, Reese Pathak, Anastasios N. Angelopoulos +2
Decision-making pipelines are generally characterized by tradeoffs among various risk functions. It is often desirable to manage such tradeoffs in a data-adaptive manner. As we dem…
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng +8
Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges. To addres…
AutoEval Done Right: Using Synthetic Data for Model Evaluation
Pierre Boyeau, Anastasios N. Angelopoulos, Nir Yosef +2
The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming. AI-labeled synthetic data can be used to decrease the number of h…
Wavefront Randomization Improves Deconvolution
Amit Kohli, Anastasios N. Angelopoulos, Laura Waller
The performance of an imaging system is limited by optical aberrations, which cause blurriness in the resulting image. Digital correction techniques, such as deconvolution, have li…
Online conformal prediction with decaying step sizes
Anastasios N. Angelopoulos, Rina Foygel Barber, Stephen Bates
We introduce a method for online conformal prediction with decaying step sizes. Like previous methods, ours possesses a retrospective guarantee of coverage for arbitrary sequences.…
PPI++: Efficient Prediction-Powered Inference
Anastasios N. Angelopoulos, John C. Duchi, Tijana Zrnic
We present PPI++: a computationally lightweight methodology for estimation and inference based on a small labeled dataset and a typically much larger dataset of machine-learning pr…
Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions
Jordan Lekeufack, Anastasios N. Angelopoulos, Andrea Bajcsy +2
We introduce Conformal Decision Theory, a framework for producing safe autonomous decisions despite imperfect machine learning predictions. Examples of such decisions are ubiquitou…
Conformal PID Control for Time Series Prediction
Anastasios N. Angelopoulos, Emmanuel J. Candes, Ryan J. Tibshirani
We study the problem of uncertainty quantification for time series prediction, with the goal of providing easy-to-use algorithms with formal guarantees. The algorithms we present b…
Class-Conditional Conformal Prediction with Many Classes
Tiffany Ding, Anastasios N. Angelopoulos, Stephen Bates +2
Standard conformal prediction methods provide a marginal coverage guarantee, which means that for a random test point, the conformal prediction set contains the true label with a u…
Prediction-Powered Inference
Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang +2
Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system.…
Label Noise Robustness of Conformal Prediction
Bat-Sheva Einbinder, Shai Feldman, Stephen Bates +3
We study the robustness of conformal prediction, a powerful tool for uncertainty quantification, to label noise. Our analysis tackles both regression and classification problems, c…
Conformal Risk Control
Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch +2
We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarante…
Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets
Charles Lu, Anastasios N. Angelopoulos, Stuart Pomerantz
The regulatory approval and broad clinical deployment of medical AI have been hampered by the perception that deep learning models fail in unpredictable and possibly catastrophic w…
Recommendation Systems with Distribution-Free Reliability Guarantees
Anastasios N. Angelopoulos, Karl Krauth, Stephen Bates +2
When building recommendation systems, we seek to output a helpful set of items to the user. Under the hood, a ranking model predicts which of two candidate items is better, and we…
Semantic uncertainty intervals for disentangled latent spaces
Swami Sankaranarayanan, Anastasios N. Angelopoulos, Stephen Bates +2
Meaningful uncertainty quantification in computer vision requires reasoning about semantic information -- say, the hair color of the person in a photo or the location of a car on t…
Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging
Anastasios N Angelopoulos, Amit P Kohli, Stephen Bates +5
Image-to-image regression is an important learning task, used frequently in biological imaging. Current algorithms, however, do not generally offer statistical guarantees that prot…
Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design
Clara Fannjiang, Stephen Bates, Anastasios N. Angelopoulos +2
Many applications of machine learning methods involve an iterative protocol in which data are collected, a model is trained, and then outputs of that model are used to choose what…
Optimal Data Selection: An Online Distributed View
Mariel Werner, Anastasios Angelopoulos, Stephen Bates +1
The blessing of ubiquitous data also comes with a curse: the communication, storage, and labeling of massive, mostly redundant datasets. We seek to solve this problem at its core,…