715 citations · 915 across the 10 of their papers we have counts for
10 papers
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini +82
AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not…
OptiSeq: Ordering Examples On-The-Fly for In-Context Learning
Rahul Atul Bhope, Praveen Venkateswaran, K. R. Jayaram +3
Developers using LLMs and LLM-based agents in their applications have provided plenty of anecdotal evidence that in-context-learning (ICL) is fragile. In this paper, we show that i…
IMBUE: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interaction
Inna Wanyin Lin, Ashish Sharma, Christopher Michael Rytting +3
Navigating certain communication situations can be challenging due to individuals' lack of skills and the interference of strong emotions. However, effective learning opportunities…
A Roadmap to Pluralistic Alignment
Taylor Sorensen, Jared Moore, Jillian Fisher +9
With increased power and prevalence of AI systems, it is ever more critical that AI systems are designed to serve all, i.e., people with diverse values and perspectives. However, a…
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay +9
Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are take…
AI Chat Assistants can Improve Conversations about Divisive Topics
Lisa P. Argyle, Ethan Busby, Joshua Gubler +4
A rapidly increasing amount of human conversation occurs online. But divisiveness and conflict can fester in text-based interactions on social media platforms, in messaging apps, a…
Leveraging Large Language Models for Multiple Choice Question Answering
Joshua Robinson, Christopher Michael Rytting, David Wingate
While large language models (LLMs) like GPT-3 have achieved impressive results on multiple choice question answering (MCQA) tasks in the zero, one, and few-shot settings, they gene…
Out of One, Many: Using Language Models to Simulate Human Samples
Lisa P. Argyle, Ethan C. Busby, Nancy Fulda +3
We propose and explore the possibility that language models can be studied as effective proxies for specific human sub-populations in social science research. Practical and researc…
An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting +6
Pre-trained language models derive substantial linguistic and factual knowledge from the massive corpora on which they are trained, and prompt engineering seeks to align these mode…
Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning
Christopher Michael Rytting, David Wingate
Large natural language models (such as GPT-3 or T5) demonstrate impressive abilities across a range of general NLP tasks. Here, we show that the knowledge embedded in such models p…