Capturing Failures of Large Language Models via Human Cognitive Biases

Abstract

Large language models generate complex, open-ended outputs: instead of outputting a class label they write summaries, generate dialogue, or produce working code. In order to asses the reliability of these open-ended generation systems, we aim to identify qualitative categories of erroneous behavior, beyond identifying individual errors. To hypothesize and test for such qualitative errors, we draw inspiration from human cognitive biases -- systematic patterns of deviation from rational judgement. Specifically, we use cognitive biases as motivation to (i) generate hypotheses for problems that models may have, and (ii) develop experiments that elicit these problems. Using code generation as a case study, we find that OpenAI's Codex errs predictably based on how the input prompt is framed, adjusts outputs towards anchors, and is biased towards outputs that mimic frequent training examples. We then use our framework to elicit high-impact errors such as incorrectly deleting files. Our results indicate that experimental methodology from cognitive science can help characterize how machine learning systems behave.

Published at NeurIPS 2022

Cited by in corpus (7)

• Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
• Harnessing Large Language Models to Enhance Self-Regulated Learning via Formative Feedback
• Mapping the individual, social, and biospheric impacts of Foundation Models
• Cognitive Effects in Large Language Models
• Decoy Effect In Search Interaction: Understanding User Behavior and Measuring System Vulnerability
• Heuristics and Biases in AI Decision-Making: Implications for Responsible AGI
• Using Artificial Populations to Study Psychological Phenomena in Neural Models