Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks

Abstract

Intuitive psychology is a pillar of common-sense reasoning. The replication of this reasoning in machine intelligence is an important stepping-stone on the way to human-like artificial intelligence. Several recent tasks and benchmarks for examining this reasoning in Large-Large Models have focused in particular on belief attribution in Theory-of-Mind tasks. These tasks have shown both successes and failures. We consider in particular a recent purported success case, and show that small variations that maintain the principles of ToM turn the results on their head. We argue that in general, the zero-hypothesis for model evaluation in intuitive psychology should be skeptical, and that outlying failure cases should outweigh average success rates. We also consider what possible future successes on Theory-of-Mind tasks by more powerful LLMs would mean for ToM tasks with people.

11 pages, 2 figures

Cited by in corpus (9)

• Large Language Models as Zero-Shot Human Models for Human-Robot Interaction
• Theory of Mind for Multi-Agent Collaboration via Large Language Models
• Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?
• Chatting with Bots: AI, Speech Acts, and the Edge of Assertion
• LLMs as Models for Analogical Reasoning
• A validity-guided workflow for robust large language model research in psychology
• From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology
• Using Artificial Populations to Study Psychological Phenomena in Neural Models
• Do Large Language Models know who did what to whom?