Gender bias and stereotypes in Large Language Models

Abstract

Large Language Models (LLMs) have made substantial progress in the past several months, shattering state-of-the-art benchmarks in many domains. This paper investigates LLMs' behavior with respect to gender stereotypes, a known issue for prior models. We use a simple paradigm to test the presence of gender bias, building on but differing from WinoBias, a commonly used gender bias dataset, which is likely to be included in the training data of current LLMs. We test four recently published LLMs and demonstrate that they express biased assumptions about men and women's occupations. Our contributions in this paper are as follows: (a) LLMs are 3-6 times more likely to choose an occupation that stereotypically aligns with a person's gender; (b) these choices align with people's perceptions better than with the ground truth as reflected in official job statistics; (c) LLMs in fact amplify the bias beyond what is reflected in perceptions or the ground truth; (d) LLMs ignore crucial ambiguities in sentence structure 95% of the time in our study items, but when explicitly prompted, they recognize the ambiguity; (e) LLMs provide explanations for their choices that are factually inaccurate and likely obscure the true reason behind their predictions. That is, they provide rationalizations of their biased behavior. This highlights a key property of these models: LLMs are trained on imbalanced datasets; as such, even with the recent successes of reinforcement learning with human feedback, they tend to reflect those imbalances back at us. As with other types of societal biases, we suggest that LLMs must be carefully tested to ensure that they treat minoritized individuals and communities equitably.

arXiv commentACM Collective Intelligence

References in corpus (13)

• Semantics derived automatically from language corpora contain human-like biases
• Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
• Release Strategies and the Social Impacts of Language Models
• Fairness Without Demographics in Repeated Loss Minimization
• How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Study
• Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity
• Getting Gender Right in Neural Machine Translation
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• Towards Measuring the Representation of Subjective Global Opinions in Language Models
• The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges
• What are the biases in my word embedding?

Cited by in corpus (27)

• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
• Large Language Models (LLMs) as Agents for Augmented Democracy
• How understanding large language models can inform the use of ChatGPT in physics education
• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• (A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice
• Data Feminism for AI
• ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?
• Conversate: Supporting Reflective Learning in Interview Practice Through Interactive Simulation and Dialogic Feedback
• The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiring
• AI-AI Bias: large language models favor communications generated by large language models
• Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigation
• Gender and content bias in Large Language Models: a case study on Google Gemini 2.0 Flash Experimental
• `My Dataset of Love': A Preliminary Mixed-Method Exploration of Human-AI Romantic Relationships
• Artificial Intelligence for Collective Intelligence: A National-Scale Research Strategy
• Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategies
• Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Models
• Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge
• A Matter of Perspective(s): Contrasting Human and LLM Argumentation in Subjective Decision-Making on Subtle Sexism
• Fairness Definitions in Language Models Explained
• LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Data
• Understanding Gender Bias in AI-Generated Product Descriptions
• LLM Bias Evaluation: Gender, Racial, and Age Disparities in Occupational and Crime Scenarios
• RV4Chatbot: Are Chatbots Allowed to Dream of Electric Sheep?
• Constella: Supporting Storywriters' Interconnected Character Creation through LLM-based Multi-Agents
• Survival of the Notable: Gender Asymmetry in Wikipedia Collective Deliberations
• From Mind Mapping to Dream Interpretation: Understanding Muslim Users' Engagement with GenAI Tools in Religious Context
• Detecting Gender Stereotypes in Scratch Programming Tutorials