Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
arXiv:2208.10264
Abstract
We introduce a new type of test, called a Turing Experiment (TE), for evaluating to what extent a given language model, such as GPT models, can simulate different aspects of human behavior. A TE can also reveal consistent distortions in a language model's simulation of a specific human behavior. Unlike the Turing Test, which involves simulating a single arbitrary individual, a TE requires simulating a representative sample of participants in human subject research. We carry out TEs that attempt to replicate well-established findings from prior studies. We design a methodology for simulating TEs and illustrate its use to compare how well different language models are able to reproduce classic economic, psycholinguistic, and social psychology experiments: Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom of Crowds. In the first three TEs, the existing findings were replicated using recent models, while the last TE reveals a "hyper-accuracy distortion" present in some language models (including ChatGPT and GPT-4), which could affect downstream applications in education and the arts.
Accepted for oral presentation at International Conference on Machine Learning (ICML) 2023
Cited by in corpus (8)
- Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations
- Emergent social conventions and collective bias in LLM populations
- The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective
- Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias
- Human-interpretable clustering of short-text using large language models
- Large language models can replicate cross-cultural differences in personality
- A validity-guided workflow for robust large language model research in psychology
- From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology