7 papers · 1 filter
Flexible Routing via Uncertainty Decomposition
Charlotte Peale, Siddartha Devic, Parikshit Gopalan +2
A key strategy for balancing performance and cost in modern machine learning systems is to dynamically route queries to either a low-cost model or a more expensive oracle (such as…
Are LLM Decisions Faithful to Verbal Confidence?
Jiawei Wang, Yanfei Zhou, Siddartha Devic +1
Large Language Models (LLMs) can produce surprisingly sophisticated estimates of their own uncertainty. However, it remains unclear to what extent this expressed confidence is tied…
Proper Learnability and the Role of Unlabeled Data
Julian Asilis, Siddartha Devic, Shaddin Dughmi +2
Proper learning refers to the setting in which learners must emit predictors in the underlying hypothesis class , and often leads to learners with simple algorithmic forms (e.g.…
Auditability and the Landscape of Distance to Multicalibration
Nathan Derhake, Siddartha Devic, Dutch Hansen +2
Calibration is a critical property for establishing the trustworthiness of predictors that provide uncertainty estimates. Multicalibration is a strengthening of calibration which r…
An Efficient Plugin Method for Metric Optimization of Black-Box Models
Siddartha Devic, Nurendra Choudhary, Anirudh Srinivasan +3
Many machine learning algorithms and classifiers are available only via API queries as a ``black-box'' -- that is, the downstream user has no ability to change, re-train, or fine-t…
When is Multicalibration Post-Processing Necessary?
Dutch Hansen, Siddartha Devic, Preetum Nakkiran +1
Calibration is a well-studied property of predictors which guarantees meaningful uncertainty estimates. Multicalibration is a related notion -- originating in algorithmic fairness…