6 papers
Language Models Are "Insecure" Reporters
Jenny Y. Huang, Jiameng Fan, Ahmed Imtiaz Humayun +5
As large language models are deployed in increasingly autonomous long-horizon tasks, manually auditing and verifying the actions, artifacts, and outputs of models becomes more diff…
Navigating the Conceptual Multiverse
Andre Ye, Jenny Y. Huang, Alicia Guo +3
When language models answer open-ended questions, they implicitly make decisions that shape their answers. However, the models do not show these decisions to the user, so users can…
Do LLMs Benefit From Their Own Words?
Jenny Y. Huang, Leshem Choshen, Wei Sun +5
In multi-turn conversations, large language models typically condition on the full conversation history: both past user prompts and assistant responses. We revisit this design choi…
Latency and Token-Aware Test-Time Compute
Jenny Y. Huang, Mehul Damani, Yousef El-Kurdi +2
Inference-time scaling has emerged as a powerful way to improve large language model (LLM) performance by generating multiple candidate responses and selecting among them. However,…
Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
Jenny Y. Huang, Yunyi Shen, Dennis Wei +1
We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a worst-case very small fraction of prefer…
Approximations to worst-case data dropping: unmasking failure modes
Jenny Y. Huang, David R. Burt, Yunyi Shen +2
A data analyst might worry about generalization if dropping a very small fraction of data points from a study could change its substantive conclusions. Checking this non-robustness…