134 citations · 242 across the 39 of their papers we have counts for
43 papers
Flexible Language-Guided Representation Editing for Adaptive Robot Behavior
Audrey Lee, Nathaniel Dennler, Ilia Sucholutsky +1
Trajectory demonstrations show robots how to behave but not why, leaving them to guess which features of the task actually matter. Incorrect guesses produce misaligned internal rep…
CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
Lance Ying, Jinzhou Wu, Yingshan Susan Wang +53
Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways…
Prioritization of Risks from Artificial Intelligence: A Delphi Study of 272 International Experts
Alexander K. Saeri, Jess Graham, Michael Noetel +185
Artificial intelligence poses many risks, ranging from familiar present-day harms to unprecedented and potentially catastrophic ones. Effective risk management requires prioritizat…
Human agency in initial human-AI proof formalization workflows
Katherine M. Collins, Simon Frieder, Jonas Bayer +14
For centuries, human mathematicians have written proofs to substantiate their mathematical arguments; yet, the ability to automatically verify the validity of proofs has long been…
Cognitive offloading and the speedup illusion in human-AI interaction
Sunny Yu, Myra Cheng, Ahmad Jabbar +4
Large language models (LLMs) have the potential to boost human productivity by speeding up task completion -- provided users know when to offload cognitive work to them. But we do…
The efficiency-gain illusion: People underestimate the rate of AI use and overestimate its benefits on simple tasks
Sunny Yu, Myra Cheng, Ahmad Jabbar +4
People are increasingly turning to AI assistance for simple tasks, e.g., arithmetic, spell-check, and answering simple questions. But does AI assistance actually save users time an…
On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradation
Andy Han, Kristina Fujimoto, Avidan Shah +5
Aligned models can misbehave in several ways: they are often sycophantic, fall victim to jailbreaks, or fail to include appropriate safety warnings. Consistency training is a promi…
Medical Model Synthesis Architectures: A Case Study
Katherine M. Collins, Marlene Berke, Ilia Sucholutsky +6
Medicine is rife with high-stakes uncertainty. Doctors routinely make clinical judgments and decisions that juggle many fundamental unknowns, like predictions about what might be c…
Improving the Efficiency of Language Agent Teams with Adaptive Task Graphs
Elizabeth Mieczkowski, Alexander Ku, Tiwalayo Eisape +5
Large language models (LLMs) are increasingly deployed in teams, yet existing coordination approaches often occupy two extremes. Highly structured methods rely on fixed roles, pipe…
Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models
Ayush Rajesh Jhaveri, Anthony GX-Chen, Ilia Sucholutsky +1
Confirmation bias, the tendency to seek evidence that supports rather than challenges one's belief, hinders one's reasoning ability. We examine whether large language models (LLMs)…
Do Large Language Models Mentalize When They Teach?
Sevan K. Harootonian, Mark K. Ho, Thomas L. Griffiths +2
How do LLMs decide what to teach next: by reasoning about a learner's knowledge, or by using simpler rules of thumb? We test this in a controlled task previously used to study huma…
Under the Influence: Quantifying Persuasion and Vigilance in Large Language Models
Sasha Robinson, Katherine M. Collins, Ilia Sucholutsky +1
With increasing integration of Large Language Models (LLMs) into areas of high-stakes human decision-making, it is important to understand the risks they introduce as advisors. To…
Language Model Teams as Distributed Systems
Elizabeth Mieczkowski, Katherine M. Collins, Ilia Sucholutsky +2
Large language models (LLMs) are growing increasingly capable, prompting recent interest in LLM teams. Yet, despite increased deployment of LLM teams at scale, we lack a principled…
Why Human Guidance Matters in Collaborative Vibe Coding
Haoyu Hu, Raja Marjieh, Katherine M Collins +4
Writing code has been one of the most transformative ways for human societies to translate abstract ideas into tangible technologies. Modern AI is changing this process by enabling…
Measuring and mitigating overreliance to build human-compatible AI
Lujain Ibrahim, Katherine M. Collins, Sunnie S. Y. Kim +14
Large language models (LLMs) distinguish themselves from previous technologies by functioning as collaborative ``thought partners,'' capable of engaging more fluidly in natural lan…
Identifying, Evaluating, and Mitigating Risks of AI Thought Partnerships
Kerem Oktar, Katherine M. Collins, Jose Hernandez-Orallo +4
Artificial Intelligence (AI) systems have historically been used as tools that execute narrowly defined tasks. Yet recent advances in AI have unlocked possibilities for a new class…
When Should We Orchestrate Multiple Agents?
Umang Bhatt, Sanyam Kapoor, Mihir Upadhyay +6
Strategies for orchestrating the interactions between multiple agents, both human and artificial, can wildly overestimate performance and underestimate the cost of orchestration. W…
Levels of Analysis for Large Language Models
Alexander Y. Ku, Declan Campbell, Xuechunzi Bai +10
Modern artificial intelligence systems, such as large language models, are increasingly powerful but also increasingly hard to understand. Recognizing this problem as analogous to…
What is a Number, That a Large Language Model May Know It?
Raja Marjieh, Veniamin Veselovsky, Thomas L. Griffiths +1
Numbers are a basic part of how humans represent and describe the world around them. As a consequence, learning effective representations of numbers is critical for the success of…
On Benchmarking Human-Like Intelligence in Machines
Lance Ying, Katherine M. Collins, Lionel Wong +6
Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated data, are increasingly posited as…
Revisiting Rogers' Paradox in the Context of Human-AI Interaction
Katherine M. Collins, Umang Bhatt, Ilia Sucholutsky
Humans learn about the world, and how to act in the world, in many ways: from individually conducting experiments to observing and reproducing others' behavior. Different learning…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1168
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…
Quantifying Knowledge Distillation Using Partial Information Decomposition
Pasan Dissanayake, Faisal Hamman, Barproda Halder +3
Knowledge distillation deploys complex machine learning models in resource-constrained environments by training a smaller student model to emulate internal representations of a com…
Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
Ryan Liu, Jiayi Geng, Addison J. Wu +3
Chain-of-thought (CoT) prompting has become a widely used strategy for improving large language and multimodal model performance. However, it is still an open question under which…