6 citations · 23 across the 44 of their papers we have counts for
51 papers · 1 filter
BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks
Nishant Balepur, Bhavya Rajasekaran, Jane Oh +7
Multiple-choice question answering (MCQA) is standard in NLP, but benchmarks lack rigorous quality control. We present BenchMarker, an education-inspired toolkit using LLM judges t…
Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations
Christabel Acquaye, Yi Ting Huang, Marine Carpuat +1
Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. We investigate the predictive value of open-source large language mod…
Arguments that Alter Minds: LLM Rationales Sway Human (and LLM) Notions of Plausibility
Shramay Palta, Peter Rankel, Sarah Wiegreffe +1
We investigate the degree to which human (and LLM) plausibility judgments of multiple-choice commonsense benchmark answers are subject to influence by (im)plausibility arguments fo…
Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers
Nishant Balepur, Atrey Desai, Rachel Rudinger
Large language models (LLMs) now give reasoning before answering, excelling in tasks like multiple-choice question answering (MCQA). Yet, a concern is that LLMs do not solve MCQs a…
A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users
Nishant Balepur, Matthew Shu, Yoo Yeon Sung +5
To assist users in complex tasks, LLMs generate plans: step-by-step instructions towards a goal. While alignment methods aim to ensure LLM plans are helpful, they train (RLHF) or e…
'Rich Dad, Poor Lad': How do Large Language Models Contextualize Socioeconomic Factors in College Admission ?
Huy Nghiem, Phuong-Anh Nguyen-Le, John Prindle +2
Large Language Models (LLMs) are increasingly involved in high-stakes domains, yet how they reason about socially sensitive decisions remains underexplored. We present a large-scal…
Multiple LLM Agents Debate for Equitable Cultural Alignment
Dayeon Ki, Rachel Rudinger, Tianyi Zhou +1
Large Language Models (LLMs) need to adapt their predictions to diverse cultural contexts to benefit diverse communities across the world. While previous efforts have focused on si…
On the Mutual Influence of Gender and Occupation in LLM Representations
Haozhe An, Connor Baumler, Abhilasha Sancheti +1
We examine LLM representations of gender for first names in various occupational contexts to study how occupations and the gender perception of first names in LLMs influence each o…
NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals
Neha Srikanth, Rachel Rudinger
Decomposition of text into atomic propositions is a flexible framework allowing for the closer inspection of input and output text. We use atomic decomposition of hypotheses in two…
Language Models Predict Empathy Gaps Between Social In-groups and Out-groups
Yu Hou, Hal Daumé, Rachel Rudinger
Studies of human psychology have demonstrated that people are more motivated to extend empathy to in-group members than out-group members (Cikara et al., 2011). In this study, we i…
Understanding Common Ground Misalignment in Goal-Oriented Dialog: A Case-Study with Ubuntu Chat Logs
Rupak Sarkar, Neha Srikanth, Taylor Hudson +3
While it is commonly accepted that maintaining common ground plays a role in conversational success, little prior research exists connecting conversational grounding to success in…
FRIDA to the Rescue! Analyzing Synthetic Data Effectiveness in Object-Based Common Sense Reasoning for Disaster Response
Mollie Shichman, Claire Bonial, Austin Blodgett +3
During Human Robot Interactions in disaster relief scenarios, Large Language Models (LLMs) have the potential for substantial physical reasoning to assist in mission objectives. Ho…
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above
Nishant Balepur, Rachel Rudinger, Jordan Lee Boyd-Graber
Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing, but we argue for its reform. We first reveal flaws in MCQA's f…
Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas
Nishant Balepur, Vishakh Padmakumar, Fumeng Yang +3
LLMs are aligned to follow input instructions by learning which of two responses users prefer for a prompt. However, such preference data do not convey why users prefer responses t…
Natural Language Inference Improves Compositionality in Vision-Language Models
Paola Cascante-Bonilla, Yu Hou, Yang Trista Cao +2
Compositional reasoning in Vision-Language Models (VLMs) remains challenging as these models often struggle to relate objects, attributes, and spatial relationships. Recent methods…
Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the U.S
Christabel Acquaye, Haozhe An, Rachel Rudinger
Recent work has highlighted the culturally-contingent nature of commonsense knowledge. We introduce AMAMMER, a test set of 525 multiple-choice questions designed to evaluate the…
Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning
Shramay Palta, Nishant Balepur, Peter Rankel +3
Questions involving commonsense reasoning about everyday situations often admit many or answers. In contrast, multiple-choice question (MCQ…
On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models
Abhilasha Sancheti, Haozhe An, Rachel Rudinger
We study the presence of heteronormative biases and prejudice against interracial romantic relationships in large language models by performing controlled name-replacement experime…
Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions
Angana Borah, Rada Mihalcea
As Large Language Models (LLMs) continue to evolve, they are increasingly being employed in numerous studies to simulate societies and execute diverse social tasks. However, LLMs a…
Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?
Nishant Balepur, Feng Gu, Abhilasha Ravichander +3
Question answering (QA), giving correct answers to questions, is a popular task, but we test reverse question answering (RQA): for an input answer, give a question with that answer…
Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?
Nishant Balepur, Rachel Rudinger
Recent work shows that large language models (LLMs) can answer multiple-choice questions using only the choices, but does this mean that MCQA leaderboard rankings of LLMs are large…
Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?
Haozhe An, Christabel Acquaye, Colin Wang +2
We examine whether large language models (LLMs) exhibit race- and gender-based name discrimination in hiring decisions, similar to classic findings in the social sciences (Bertrand…
How often are errors in natural language reasoning due to paraphrastic variability?
Neha Srikanth, Marine Carpuat, Rachel Rudinger
Large language models have been shown to behave inconsistently in response to meaning-preserving paraphrastic inputs. At the same time, researchers evaluate the knowledge and reaso…
Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?
Nishant Balepur, Abhilasha Ravichander, Rachel Rudinger
Multiple-choice question answering (MCQA) is often used to evaluate large language models (LLMs). To see if MCQA assesses LLMs as intended, we probe if LLMs can perform MCQA with c…