1 citations · 1 across the 8 of their papers we have counts for
7 papers · 1 filter
Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting
Siyu Zhou, Jialin Ouyang, Fan Xia +8
Continuous monitoring of generative AI systems requires detecting distribution shifts in the unstructured data they ingest and generate. Existing monitoring methods typically contr…
Testing For Distribution Shifts with Conditional Conformal Test Martingales
Shalev Shaer, Yarin Bar, Drew Prinster +1
We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existin…
E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
Shuvom Sadhuka, Drew Prinster, Clara Fannjiang +4
Agentic AI systems execute a sequence of actions, such as reasoning steps or tool calls, in response to a user prompt. To evaluate the success of their trajectories, researchers ha…
Improving Coverage in Combined Prediction Sets with Weighted p-values
Gina Wong, Drew Prinster, Suchi Saria +2
Conformal prediction quantifies the uncertainty of machine learning models by augmenting point predictions with valid prediction sets. For complex scenarios involving multiple tria…
WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales
Drew Prinster, Xing Han, Anqi Liu +1
Responsibly deploying artificial intelligence (AI) / machine learning (ML) systems in high-stakes settings arguably requires not only proof of system reliability, but also continua…
Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)
Drew Prinster, Samuel Stanton, Anqi Liu +1
As artificial intelligence (AI) / machine learning (ML) gain widespread adoption, practitioners are increasingly seeking means to quantify and control the risk these systems incur.…
JAWS: Auditing Predictive Uncertainty Under Covariate Shift
Drew Prinster, Anqi Liu, Suchi Saria
We propose \textbf{JAWS}, a series of wrapper methods for distribution-free uncertainty quantification tasks under covariate shift, centered on the core method \textbf{JAW}, the \t…