3 citations · 3 across the 9 of their papers we have counts for
10 papers
The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge
Alexander Gill, Md Farhan Ishmam, Xuyen Nguyen +6
Existing computer-use agent benchmarks do not fully evaluate agents acting as assistants. A useful assistant retrieves information across complex, multi-step workflows, synthesizes…
Teaching People LLM's Errors and Getting it Right
Nathan Stringham, Fateme Hashemi Chaleshtori, Xinyuan Yan +3
People use large language models (LLMs) when they should not. This is partly because they see LLMs compose poems and answer intricate questions, so they understandably, but incorre…
MixAssist: An Audio-Language Dataset for Co-Creative AI Assistance in Music Mixing
Michael Clemens, Ana Marasović
While AI presents significant potential for enhancing music mixing and mastering workflows, current research predominantly emphasizes end-to-end automation or generation, often ove…
BriefMe: A Legal NLP Benchmark for Assisting with Legal Briefs
Jesse Woo, Fateme Hashemi Chaleshtori, Ana Marasović +1
A core part of legal work that has been under-explored in Legal NLP is the writing and editing of legal briefs. This requires not only a thorough understanding of the law of a juri…
What Has Been Lost with Synthetic Evaluation?
Alexander Gill, Abhilasha Ravichander, Ana Marasović
Large language models (LLMs) are increasingly used for data generation. However, creating evaluation benchmarks raises the bar for this emerging paradigm. Benchmarks must target sp…
Societal Alignment Frameworks Can Improve LLM Alignment
Karolina Stańczak, Nicholas Meade, Mehar Bhatia +14
Recent progress in large language models (LLMs) has focused on producing responses that meet human expectations and align with shared values - a process coined alignment. However,…
Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps
Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasović +1
When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction.…
Chain-of-Thought Unfaithfulness as Disguised Accuracy
Oliver Bentham, Nathan Stringham, Ana Marasović
Understanding the extent to which Chain-of-Thought (CoT) generations align with a large language model's (LLM) internal computations is critical for deciding whether to trust an LL…
Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness
Ashim Gupta, Rishanth Rajendhran, Nathan Stringham +2
Do larger and more performant models resolve NLP's longstanding robustness issues? We investigate this question using over 20 models of different sizes spanning different architect…
How Much Consistency Is Your Accuracy Worth?
Jacob K. Johnson, Ana Marasović
Contrast set consistency is a robustness measurement that evaluates the rate at which a model correctly responds to all instances in a bundle of minimally different examples relyin…