5 papers · 1 filter
Testing For Distribution Shifts with Conditional Conformal Test Martingales
Shalev Shaer, Yarin Bar, Drew Prinster +1
We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existin…
E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
Shuvom Sadhuka, Drew Prinster, Clara Fannjiang +4
Agentic AI systems execute a sequence of actions, such as reasoning steps or tool calls, in response to a user prompt. To evaluate the success of their trajectories, researchers ha…
Improving Coverage in Combined Prediction Sets with Weighted p-values
Gina Wong, Drew Prinster, Suchi Saria +2
Conformal prediction quantifies the uncertainty of machine learning models by augmenting point predictions with valid prediction sets. For complex scenarios involving multiple tria…
WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales
Drew Prinster, Xing Han, Anqi Liu +1
Responsibly deploying artificial intelligence (AI) / machine learning (ML) systems in high-stakes settings arguably requires not only proof of system reliability, but also continua…
Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)
Drew Prinster, Samuel Stanton, Anqi Liu +1
As artificial intelligence (AI) / machine learning (ML) gain widespread adoption, practitioners are increasingly seeking means to quantify and control the risk these systems incur.…