GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations

Abstract

Large pre-trained language models have become a crucial backbone for many downstream tasks in natural language processing (NLP), and while they are trained on a plethora of data containing a variety of biases, such as gender biases, it has been shown that they can also inherit such biases in their weights, potentially affecting their prediction behavior. However, it is unclear to what extent these biases also affect feature attributions generated by applying "explainable artificial intelligence" (XAI) techniques, possibly in unfavorable ways. To systematically study this question, we create a gender-controlled text dataset, GECO, in which the alteration of grammatical gender forms induces class-specific words and provides ground truth feature attributions for gender classification tasks. This enables an objective evaluation of the correctness of XAI methods. We apply this dataset to the pre-trained BERT model, which we fine-tune to different degrees, to quantitatively measure how pre-training induces undesirable bias in feature attributions and to what extent fine-tuning can mitigate such explanation bias. To this extent, we provide GECOBench, a rigorous quantitative evaluation framework for benchmarking popular XAI methods. We show a clear dependency between explanation performance and the number of fine-tuned layers, where XAI methods are observed to benefit particularly from fine-tuning or complete retraining of embedding layers.

Published in Frontiers

References in corpus (11)

• Interpretable machine learning: definitions, methods, and applications
• Captum: A unified and generic model interpretability library for PyTorch
• Learning the Difference that Makes a Difference with Counterfactually-Augmented Data
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• First Women, Second Sex: Gender Bias in Wikipedia
• The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task
• Benchmarking the Influence of Pre-training on Explanation Performance in MR Image Classification
• Explaining the data or explaining a model? Shapley values that uncover non-linear dependencies
• QUACKIE: A NLP Classification Task With Ground Truth Explanations