A validity-guided workflow for robust large language model research in psychology

Abstract

Large language models (LLMs) are rapidly being integrated into psychological and behavioral research as research tools, evaluation targets, human simulators, and cognitive models. Yet recent evidence reveals severe measurement unreliability: personality assessments degenerate under factor analysis, moral preferences reverse with punctuation changes, and theory-of-mind accuracy varies widely with trivial rephrasing. These "measurement phantoms"--statistical artifacts masquerading as psychological phenomena--threaten the validity of a growing body of research. Guided by the dual-validity framework that integrates psychometrics with causal inference, we present a six-stage workflow that scales validity requirements to research ambition--using LLMs to code text requires basic reliability and accuracy, whereas claims about psychological properties demand comprehensive construct validation. Researchers must (1) explicitly define their research goal and corresponding validity requirements, (2) develop and validate computational instruments through psychometric testing, (3) design experiments that control for computational confounds, (4) execute protocols transparently, (5) analyze data with methods appropriate for non-independent observations, and (6) report findings within boundaries and use results to refine theory. We illustrate the workflow through an example of model evaluation--"LLM selfhood"--showing how systematic validation can distinguish genuine computational phenomena from measurement artifacts. By establishing validated computational instruments and transparent practices, this workflow provides a path toward building a robust empirical foundation for AI psychology research.

References in corpus (23)

• Using cognitive psychology to understand GPT-3
• Evaluating Large Language Models in Theory of Mind Tasks
• How to avoid machine learning pitfalls: a guide for academic researchers
• Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
• Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks
• Evaluating Large Language Models in Analysing Classroom Dialogue
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
• The Moral Machine Experiment on Large Language Models
• Personality Traits in Large Language Models
• The Capability of Large Language Models to Measure Psychiatric Functioning
• Who is GPT-3? An Exploration of Personality, Values and Demographics
• A Survey on Data Contamination for Large Language Models
• Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement
• Cognitive phantoms in LLMs through the lens of latent variables
• Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
• Challenging the Validity of Personality Tests for Large Language Models
• Can LLM "Self-report"?: Evaluating the Validity of Self-report Scales in Measuring Personality Design in LLM-based Chatbots
• From traces to measures: Large language models as a tool for psychological measurement from text
• Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics
• Beyond BFI: The CSI for Enhanced Reliability and Validity in Evaluating LLM Personality Traits
• Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
• Multi-ToM: Evaluating Multilingual Theory of Mind Capabilities in Large Language Models

Cited by in corpus (2)

• Large Language Models as Psychological Simulators: A Methodological Guide
• From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology