Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Tasks

Abstract

The rapid deployment of artificial intelligence (AI) models demands a thorough investigation of biases and risks inherent in these models to understand their impact on individuals and society. This study extends the focus of bias evaluation in extant work by examining bias against social stigmas on a large scale. It focuses on 93 stigmatized groups in the United States, including a wide range of conditions related to disease, disability, drug use, mental illness, religion, sexuality, socioeconomic status, and other relevant factors. We investigate bias against these groups in English pre-trained Masked Language Models (MLMs) and their downstream sentiment classification tasks. To evaluate the presence of bias against 93 stigmatized conditions, we identify 29 non-stigmatized conditions to conduct a comparative analysis. Building upon a psychology scale of social rejection, the Social Distance Scale, we prompt six MLMs: RoBERTa-base, RoBERTa-large, XLNet-large, BERTweet-base, BERTweet-large, and DistilBERT. We use human annotations to analyze the predicted words from these models, with which we measure the extent of bias against stigmatized groups. When prompts include stigmatized conditions, the probability of MLMs predicting negative words is approximately 20 percent higher than when prompts have non-stigmatized conditions. In the sentiment classification tasks, when sentences include stigmatized conditions related to diseases, disability, education, and mental illness, they are more likely to be classified as negative. We also observe a strong correlation between bias in MLMs and their downstream sentiment classifiers (r =0.79). The evidence indicates that MLMs and their downstream sentiment classification tasks exhibit biases against socially stigmatized groups.

arXiv comment20 pages,12 figures,2 tables; ACM FAccT 2023

References in corpus (5)

• Semantics derived automatically from language corpora contain human-like biases
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases
• Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task
• Reducing Sentiment Bias in Language Models via Counterfactual Evaluation

Cited by in corpus (8)

• Identifying and Improving Disability Bias in GPT-Based Resume Screening
• Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
• Evaluating Biased Attitude Associations of Language Models in an Intersectional Context
• The Power of Absence: Thinking with Archival Theory in Algorithmic Design
• Laissez-Faire Harms: Algorithmic Biases in Generative Language Models
• Not Like Us, Hunty: Measuring Perceptions and Behavioral Effects of Minoritized Anthropomorphic Cues in LLMs
• Exploring LGBTQ+ Bias in Generative AI Answers across Different Country and Religious Contexts
• Shiny Stories, Hidden Struggles: Investigating the Representation of Disability Through the Lens of LLMs