6 papers
A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge
Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6
Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human e…
On the Convergence of Multicalibration Gradient Boosting
Daniel Haimovich, Fridolin Linder, Lorenzo Perini +2
Multicalibration gradient boosting has recently emerged as a scalable method that empirically produces approximately multicalibrated predictors and has been deployed at web scale.…
Multicalibration Yields Better Matchings
Riccardo Colini Baldeschi, Simone Di Gregorio, Simone Fioravanti +9
Consider the problem of finding the best matching in a weighted graph where we only have access to predictions of the actual stochastic weights, based on an underlying context. If…
Multiclass Local Calibration with the Jensen-Shannon Distance
Cesare Barbera, Lorenzo Perini, Giovanni De Toni +2
Developing trustworthy Machine Learning (ML) models requires their predicted probabilities to be well-calibrated, meaning they should reflect true-class frequencies. Among calibrat…
MCGrad: Multicalibration at Web Scale
Niek Tax, Lorenzo Perini, Fridolin Linder +5
We propose MCGrad, a novel and scalable multicalibration algorithm. Multicalibration - calibration in subgroups of the data - is an important property for the performance of machin…
Measuring multi-calibration
Ido Guy, Daniel Haimovich, Fridolin Linder +4
A suitable scalar metric can help measure multi-calibration, defined as follows. When the expected values of observed responses are equal to corresponding predicted probabilities,…