Measuring and Reducing Gendered Correlations in Pre-trained Models

Abstract

Pre-trained models have revolutionized natural language understanding. However, researchers have found they can encode artifacts undesired in many applications, such as professions correlating with one gender more than another. We explore such gendered correlations as a case study for how to address unintended correlations in pre-trained models. We define metrics and reveal that it is possible for models with similar accuracy to encode correlations at very different rates. We show how measured correlations can be reduced with general-purpose techniques, and highlight the trade offs different strategies have. With these results, we make recommendations for training robust models: (1) carefully evaluate unintended correlations, (2) be mindful of seemingly innocuous configuration differences, and (3) focus on general mitigations.

References in corpus (16)

• Distributed Representations of Words and Phrases and their Compositionality
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Semantics derived automatically from language corpora contain human-like biases
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them
• Censoring Representations with an Adversary
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Assessing Social and Intersectional Biases in Contextualized Word Representations
• What's in a Name? Reducing Bias in Bios without Access to Protected Attributes
• Learning Gender-Neutral Word Embeddings
• Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns

Cited by in corpus (10)

• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Auditing large language models: a three-layered approach
• Rethinking Search: Making Domain Experts out of Dilettantes
• How Reliable are Model Diagnostics?
• Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models
• From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Models
• Undesirable Biases in NLP: Addressing Challenges of Measurement
• SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets
• Gender Bias in Multilingual Neural Machine Translation: The Architecture Matters
• Evaluating Company-specific Biases in Financial Sentiment Analysis using Large Language Models