8 papers
Flexible Routing via Uncertainty Decomposition
Charlotte Peale, Siddartha Devic, Parikshit Gopalan +2
A key strategy for balancing performance and cost in modern machine learning systems is to dynamically route queries to either a low-cost model or a more expensive oracle (such as…
Are LLM Decisions Faithful to Verbal Confidence?
Jiawei Wang, Yanfei Zhou, Siddartha Devic +1
Large Language Models (LLMs) can produce surprisingly sophisticated estimates of their own uncertainty. However, it remains unclear to what extent this expressed confidence is tied…
Proper Learnability and the Role of Unlabeled Data
Julian Asilis, Siddartha Devic, Shaddin Dughmi +2
Proper learning refers to the setting in which learners must emit predictors in the underlying hypothesis class , and often leads to learners with simple algorithmic forms (e.g.…
An External Fairness Evaluation of LinkedIn Talent Search
Tina Behzad, Siddartha Devic, Vatsal Sharan +2
We conduct an independent, third-party audit for bias of LinkedIn's Talent Search ranking system, focusing on potential ranking bias across two attributes: gender and race. To do s…
Trace Length is a Simple Uncertainty Signal in Reasoning Models
Siddartha Devic, Charlotte Peale, Arwen Bradley +3
Uncertainty quantification for LLMs is a key research direction towards addressing hallucination and other issues that limit their reliable deployment. In this work, we show that r…
Auditability and the Landscape of Distance to Multicalibration
Nathan Derhake, Siddartha Devic, Dutch Hansen +2
Calibration is a critical property for establishing the trustworthiness of predictors that provide uncertainty estimates. Multicalibration is a strengthening of calibration which r…