Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior
arXiv:2407.02099 · doi:10.1371/journal.pone.0325664
Abstract
One way to personalize and steer generations from large language models (LLM) is to assign a persona: a role that describes how the user expects the LLM to behave (e.g., a helpful assistant, a teacher, a woman). This paper investigates how personas affect diverse aspects of model behavior. We assign to seven LLMs 162 personas from 12 categories spanning variables like gender, sexual orientation, and occupation. We prompt them to answer questions from five datasets covering objective (e.g., questions about math and history) and subjective tasks (e.g., questions about beliefs and values). We also compare persona's generations to two baseline settings: a control persona setting with 30 paraphrases of "a helpful assistant" to control for models' prompt sensitivity, and an empty persona setting where no persona is assigned. We find that for all models and datasets, personas show greater variability than the control setting and that some measures of persona behavior generalize across models.
20 pages, 12 figures. Accepted at PLOS One
References in corpus (13)
- Out of One, Many: Using Language Models to Simulate Human Samples
- Measuring Massive Multitask Language Understanding
- Gemma: Open Models Based on Gemini Research and Technology
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
- Zephyr: Direct Distillation of LM Alignment
- Evaluating and Inducing Personality in Pre-trained Language Models
- In-Context Impersonation Reveals Large Language Models' Strengths and Biases
- Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
- Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs
- Large Language Models as Superpositions of Cultural Perspectives
- Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems
- CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations
- Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset