Adapt or Calibrate? How Local Labels Buy Certified Coverage in Medical VQA

Abstract

An answer from a vision-language model (VLM) to a medical image question can be accepted without review only under a threshold bounding the error of accepted answers, and certifying that bound costs local labels. We ask what such labels can certify for nine confidence estimators on five general-purpose open-weight VLMs and three medical VQA benchmarks under Learn-then-Test, with the question as the exchangeable unit. Estimators trained on natural images transfer poorly: the best pooled certificate covers 11.5% of SLAKE answers at a 20% error tolerance and nothing on PATH-VQA. With 1,000 labeled SLAKE questions, adapting a probe on half of them raises pooled certified coverage from 6.8% to 37.6% (0.0% to 30.6% with the image as the unit), above the best threshold-only estimator (11.3%). On PATH-VQA, adaptation certifies 11.4% with 2,500 labels; thresholds alone certify nothing. Yet for strong single models, small budgets are better spent on the threshold. A question-only classifier recovers much of adaptation's ranking gain but certifies far less. Neither procedure yields a pooled certificate at 10%, and nothing is certified at 5%. A blank image removes 61% of correct answers, yet on GMAI-MMBench no trained probe is less confident on answers unchanged by image replacement.