Using cognitive psychology to understand GPT-3

Abstract

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. These results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.

References in corpus (4)

• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study
• Environmental drivers of systematicity and generalization in a situated agent
• Towards Understanding How Machines Can Learn Causal Overhypotheses

Cited by in corpus (27)

• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
• The Debate Over Understanding in AI's Large Language Models
• Thinking Fast and Slow in Large Language Models
• Human-Like Intuitive Behavior and Reasoning Biases Emerged in Language Models -- and Disappeared in GPT-4
• On the Creativity of Large Language Models
• Playing repeated games with Large Language Models
• (Ir)rationality and Cognitive Biases in Large Language Models
• Automating psychological hypothesis generation with AI: when large language models meet causal graph
• Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review
• Gromov-Wasserstein unsupervised alignment reveals structural correspondences between the color similarity structures of humans and large language models
• Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality
• Human-like object concept representations emerge naturally in multimodal large language models
• Six Fallacies in Substituting Large Language Models for Human Participants
• Revealing emergent human-like conceptual representations from language prediction
• Cognitive Effects in Large Language Models
• Stick to your Role! Stability of Personal Values Expressed in Large Language Models
• Cognitive phantoms in LLMs through the lens of latent variables
• Can AI with High Reasoning Ability Replicate Human-like Decision Making in Economic Experiments?
• A validity-guided workflow for robust large language model research in psychology
• Large Language Models as Psychological Simulators: A Methodological Guide
• From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology
• (Ir)rationality in AI: State of the Art, Research Challenges and Open Questions
• The meaning of prompts and the prompts of meaning: Semiotic reflections and modelling
• AI agents can coordinate beyond human scale
• MeTHanol: Modularized Thinking Language Models with Intermediate Layer Thinking, Decoding and Bootstrapping Reasoning
• Influence of Solution Efficiency and Valence of Instruction on Additive and Subtractive Solution Strategies in Humans and GPT-4
• Towards a Psychology of Machines: Large Language Models Predict Human Memory