Thinking Fast and Slow in Large Language Models

Abstract

Large language models (LLMs) are currently at the forefront of intertwining AI systems with human communication and everyday life. Therefore, it is of great importance to evaluate their emerging abilities. In this study, we show that LLMs like GPT-3 exhibit behavior that strikingly resembles human-like intuition - and the cognitive errors that come with it. However, LLMs with higher cognitive capabilities, in particular ChatGPT and GPT-4, learned to avoid succumbing to these errors and perform in a hyperrational manner. For our experiments, we probe LLMs with the Cognitive Reflection Test (CRT) as well as semantic illusions that were originally designed to investigate intuitive decision-making in humans. Our study demonstrates that investigating LLMs with methods from psychology has the potential to reveal otherwise unknown emergent traits.

References in corpus (2)

• Training language models to follow instructions with human feedback
• Using cognitive psychology to understand GPT-3

Cited by in corpus (17)

• Evaluating Large Language Models in Theory of Mind Tasks
• Deception Abilities Emerged in Large Language Models
• (Ir)rationality and Cognitive Biases in Large Language Models
• Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review
• Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing
• Human-like object concept representations emerge naturally in multimodal large language models
• Six Fallacies in Substituting Large Language Models for Human Participants
• Psychomatics -- A Multidisciplinary Framework for Understanding Artificial Minds
• Revealing emergent human-like conceptual representations from language prediction
• Can AI with High Reasoning Ability Replicate Human-like Decision Making in Economic Experiments?
• Heuristics and Biases in AI Decision-Making: Implications for Responsible AGI
• Are Large Language Models Aligned with People's Social Intuitions for Human-Robot Interactions?
• Artificial Intelligence (AI) and the Relationship between Agency, Autonomy, and Moral Patiency
• (Ir)rationality in AI: State of the Art, Research Challenges and Open Questions
• Influence of Solution Efficiency and Valence of Instruction on Additive and Subtractive Solution Strategies in Humans and GPT-4
• Addressing Moral Uncertainty using Large Language Models for Ethical Decision-Making
• Cross-Lingual Probing and Community-Grounded Analysis of Gender Bias in Low-Resource Bengali