1 citations · 1 across the 8 of their papers we have counts for
10 papers
Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting
Siyu Zhou, Jialin Ouyang, Fan Xia +8
Continuous monitoring of generative AI systems requires detecting distribution shifts in the unstructured data they ingest and generate. Existing monitoring methods typically contr…
Monitoring for Shifts at Unknown Times: Parameter-Free Conditional Conformal Martingales
Tuo Liu, Drew Prinster, Francesco Orabona
Conditional conformal test martingales provide sequential tests for detecting distribution shifts, using a fixed reference dataset to avoid contamination from test-time samples. Pr…
Toward Calibrated Mixture-of-Experts Under Distribution Shift
Gina Wong, Drew Prinster, Suchi Saria +2
Calibration aligns a model's predictive uncertainty with the frequencies of its empirical outcomes and is important for understanding and trusting reported probabilities. Recent wo…
Conformal Policy Control
Drew Prinster, Clara Fannjiang, Ji Won Park +4
An agent must try new behaviors to explore and improve. In high-stakes environments, an agent that violates safety constraints may cause harm and must be taken offline, curtailing…
Testing For Distribution Shifts with Conditional Conformal Test Martingales
Shalev Shaer, Yarin Bar, Drew Prinster +1
We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existin…
E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
Shuvom Sadhuka, Drew Prinster, Clara Fannjiang +4
Agentic AI systems execute a sequence of actions, such as reasoning steps or tool calls, in response to a user prompt. To evaluate the success of their trajectories, researchers ha…
Improving Coverage in Combined Prediction Sets with Weighted p-values
Gina Wong, Drew Prinster, Suchi Saria +2
Conformal prediction quantifies the uncertainty of machine learning models by augmenting point predictions with valid prediction sets. For complex scenarios involving multiple tria…
WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales
Drew Prinster, Xing Han, Anqi Liu +1
Responsibly deploying artificial intelligence (AI) / machine learning (ML) systems in high-stakes settings arguably requires not only proof of system reliability, but also continua…
Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)
Drew Prinster, Samuel Stanton, Anqi Liu +1
As artificial intelligence (AI) / machine learning (ML) gain widespread adoption, practitioners are increasingly seeking means to quantify and control the risk these systems incur.…
JAWS: Auditing Predictive Uncertainty Under Covariate Shift
Drew Prinster, Anqi Liu, Suchi Saria
We propose \textbf{JAWS}, a series of wrapper methods for distribution-free uncertainty quantification tasks under covariate shift, centered on the core method \textbf{JAW}, the \t…