Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias

Abstract

Common methods for interpreting neural models in natural language processing typically examine either their structure or their behavior, but not both. We propose a methodology grounded in the theory of causal mediation analysis for interpreting which parts of a model are causally implicated in its behavior. It enables us to analyze the mechanisms by which information flows from input to output through various model components, known as mediators. We apply this methodology to analyze gender bias in pre-trained Transformer language models. We study the role of individual neurons and attention heads in mediating gender bias across three datasets designed to gauge a model's sensitivity to gender bias. Our mediation analysis reveals that gender bias effects are (i) sparse, concentrated in a small part of the network; (ii) synergistic, amplified or repressed by different components; and (iii) decomposable into effects flowing directly from the input and indirectly through the mediators.

arXiv commentExpanded version

References in corpus (16)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Semantics derived automatically from language corpora contain human-like biases
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• Counterfactual Fairness
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
• Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
• Assessing Social and Intersectional Biases in Contextualized Word Representations
• Fairness Through Awareness
• Gender Bias in Contextualized Word Embeddings
• What's in a Name? Reducing Bias in Bios without Access to Protected Attributes
• Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
• Non-monotone Submodular Maximization in Exponentially Fewer Iterations
• Non-monotone Submodular Maximization with Nearly Optimal Adaptivity and Query Complexity

Cited by in corpus (19)

• On the Opportunities and Risks of Foundation Models
• Towards Continual Knowledge Learning of Language Models
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• Probing for the Usage of Grammatical Number
• iSEA: An Interactive Pipeline for Semantic Error Analysis of NLP Models
• Mediators in Determining what Processing BERT Performs First
• An Interpretability Illusion for BERT
• Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation
• Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
• Cross-Modal Conceptualization in Bottleneck Models
• Human Trajectory Prediction via Counterfactual Analysis
• Inducing Causal Structure for Interpretable Neural Networks
• Self-conditioning pre-trained language models
• What do writing features tell us about AI papers?
• On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation
• What if This Modified That? Syntactic Interventions via Counterfactual Embeddings
• Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference
• CFL: Causally Fair Language Models Through Token-level Attribute Controlled Generation
• Fine-grained Interpretation and Causation Analysis in Deep NLP Models