17 papers
The Effect of Perceived Race and Gender on Police Language Use: Experimental Evidence from VR Simulations
Sandra C. Sandoval, Navita Goyal, Rashawn Ray +3
Against the backdrop of violence in police interactions with the U.S. public, we explore how deferentially police officers speak to virtual characters depicted as Black adult males…
Arguments that Alter Minds: LLM Rationales Sway Human (and LLM) Notions of Plausibility
Shramay Palta, Peter Rankel, Sarah Wiegreffe +1
We investigate the degree to which human (and LLM) plausibility judgments of multiple-choice commonsense benchmark answers are subject to influence by (im)plausibility arguments fo…
BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks
Nishant Balepur, Bhavya Rajasekaran, Jane Oh +7
Multiple-choice question answering (MCQA) is standard in NLP, but benchmarks lack rigorous quality control. We present BenchMarker, an education-inspired toolkit using LLM judges t…
Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations
Christabel Acquaye, Yi Ting Huang, Marine Carpuat +1
Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. We investigate the predictive value of open-source large language mod…
Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers
Nishant Balepur, Atrey Desai, Rachel Rudinger
Large language models (LLMs) now give reasoning before answering, excelling in tasks like multiple-choice question answering (MCQA). Yet, a concern is that LLMs do not solve MCQs a…
A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users
Nishant Balepur, Matthew Shu, Yoo Yeon Sung +5
To assist users in complex tasks, LLMs generate plans: step-by-step instructions towards a goal. While alignment methods aim to ensure LLM plans are helpful, they train (RLHF) or e…