565 citations · 688 across the 26 of their papers we have counts for
27 papers
CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
Lance Ying, Jinzhou Wu, Yingshan Susan Wang +53
Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways…
AI Assistants Overassist
Verona Teo, Raghav Jain, Tobias Gerstenberg +1
Large language models (LLMs) are increasingly used as tutors and thought partners, helping users reason through problems. While guidance from AI assistants can scaffold thinking an…
Effective Explanations Support Planning Under Uncertainty
Hanqi Zhou, Britt Besch, Charley M. Wu +1
Explaining how to get from A to B can be challenging. It requires mentally simulating what the listener will do based on what they are told. To capture this process, we propose a c…
Why Someone Asked "Why": Foil Inference in Human and LLM Question Interpretation
Britt Besch, Tobias Gerstenberg
Explanations are inherently contrastive: E happened rather than E' because of C rather than C'. However, these contrasts, or "foils", are rarely mentioned explicitly but have to be…
From Retrieving Information to Reasoning with AI: Exploring Different Interaction Modalities to Support Human-AI Coordination in Clinical Decision-Making
Behnam Rahdari, Sameer Shaikh, Jonathan H Chen +2
LLMs are popular among clinicians for decision-support because of simple text-based interaction. However, their impact on clinicians' performance is ambiguous. Not knowing how clin…
Spot The Ball: A Benchmark for Visual Social Inference
Neha Balamurugan, Sarah Wu, Adam Chun +3
Humans excel at visual social inference, the ability to infer hidden elements of a scene from subtle behavioral cues such as other people's gaze, pose, and orientation. This abilit…
Understanding Human Limits in Pattern Recognition: A Computational Model of Sequential Reasoning in Rock, Paper, Scissors
Logan Cross, Erik Brockbank, Tobias Gerstenberg +3
How do we predict others from patterns in their behavior and what are the computational constraints that limit this ability? We investigate these questions by modeling human behavi…
Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models
Lionel Wong, Katherine M. Collins, Lance Ying +8
When faced with novel situations, people are able to marshal relevant considerations from a wide range of background knowledge and put these to use in inferences and predictions. W…
Causal-PIK: Causality-based Physical Reasoning with a Physics-Informed Kernel
Carlota Parés-Morlans, Michelle Yi, Claire Chen +4
Tasks that involve complex interactions between objects with unknown dynamics make planning before execution difficult. These tasks require agents to iteratively improve their acti…
A Communication-First Account of Explanation
Jacqueline Harding, Tobias Gerstenberg, Thomas Icard
This paper develops a formal account of causal explanation, grounded in a theory of conversational pragmatics, and inspired by the interventionist idea that explanation is about as…
Imagining and building wise machines: The centrality of AI metacognition
Samuel G. B. Johnson, Amir-Hossein Karimi, Yoshua Bengio +8
Although AI has become increasingly smart, its wisdom has not kept pace. In this article, we examine what is known about human wisdom and sketch a vision of its AI counterpart. We…
MARPLE: A Benchmark for Long-Horizon Inference
Emily Jin, Zhuoyi Huang, Jan-Philipp Fränken +7
Reconstructing past events requires reasoning across long time horizons. To figure out what happened, we need to use our prior knowledge about the world and human behavior and draw…
Human-like Affective Cognition in Foundation Models
Kanishk Gandhi, Zoe Lynch, Jan-Philipp Fränken +5
Understanding emotions is fundamental to human interaction and experience. Humans easily infer emotions from situations or facial expressions, situations from emotions, and do a va…
PERSONA: A Reproducible Testbed for Pluralistic Alignment
Louis Castricato, Nathan Lile, Rafael Rafailov +2
The rapid advancement of language models (LMs) necessitates robust alignment with diverse user values. However, current preference optimization approaches often fail to capture the…
To Err is Robotic: Rapid Value-Based Trial-and-Error during Deployment
Maximilian Du, Alexander Khazatsky, Tobias Gerstenberg +1
When faced with a novel scenario, it can be hard to succeed on the first attempt. In these challenging situations, it is important to know how to retry quickly and meaningfully. Re…
Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels
Jan-Philipp Fränken, Eric Zelikman, Rafael Rafailov +3
When prompting a language model (LM), users often expect the model to adhere to a set of behavioral principles across diverse tasks, such as producing insightful content while avoi…
Procedural Dilemma Generation for Evaluating Moral Reasoning in Humans and Language Models
Jan-Philipp Fränken, Kanishk Gandhi, Tori Qiu +3
As AI systems like language models are increasingly integrated into decision-making processes affecting people's lives, it's critical to ensure that these systems have sound moral…
STaR-GATE: Teaching Language Models to Ask Clarifying Questions
Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg +1
When prompting language models to complete a task, users often leave important aspects unsaid. While asking questions could resolve this ambiguity (GATE; Li et al., 2023), models o…
MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks
Allen Nie, Yuhui Zhang, Atharva Amdekar +3
Human commonsense understanding of the physical and social world is organized around intuitive theories. These theories support making causal and moral judgments. When something ba…
Social Contract AI: Aligning AI Assistants with Implicit Group Norms
Jan-Philipp Fränken, Sam Kwok, Peixuan Ye +6
We explore the idea of aligning an AI assistant by inverting a model of users' (unknown) preferences from observed interactions. To validate our proposal, we run proof-of-concept s…
Understanding Social Reasoning in Language Models with Language Models
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg +1
As Large Language Models (LLMs) become increasingly integrated into our everyday lives, understanding their ability to comprehend human mental states becomes critical for ensuring…
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin +3
Prior work has identified a resilient phenomenon that threatens the performance of human-AI decision-making teams: overreliance, when people agree with an AI, even when it is incor…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…
Uncalibrated Models Can Improve Human-AI Collaboration
Kailas Vodrahalli, Tobias Gerstenberg, James Zou
In many practical applications of AI, an AI model is used as a decision aid for human users. The AI provides advice that a human (sometimes) incorporates into their decision-making…