4 papers
Decomposing LLM-Judge Uncertainty to Target Expert Labels
Ryan Lail
An LLM judge evaluates outputs at scale. Experts should label only where it is least sure. Its natural escalation signal conflates two uncertainties: aleatoric, real disagreement i…
One note in three: a verified census of three deployed AI scribes, and the instrument that counted it
Sebastian Fox, Luke Markham, Ryan Lail +1
Ambient AI scribes draft clinical notes under the reassurance that a clinician signs every note. We audited three commercial AI scribes on the same 142 consultations: 565 notes fro…
LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
Sebastian Fox, Luke Markham, Ryan Lail +1
Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standa…
On Cost-Effective LLM-as-a-Judge Improvement Techniques
Ryan Lail, Luke Markham
Using a language model to score or rank candidate responses has become a scalable alternative to human evaluation in reinforcement learning from human feedback (RLHF) pipelines, be…