FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders

Abstract

Pretrained text encoders, such as BERT, have been applied increasingly in various natural language processing (NLP) tasks, and have recently demonstrated significant performance gains. However, recent studies have demonstrated the existence of social bias in these pretrained NLP models. Although prior works have made progress on word-level debiasing, improved sentence-level fairness of pretrained encoders still lacks exploration. In this paper, we proposed the first neural debiasing method for a pretrained sentence encoder, which transforms the pretrained encoder outputs into debiased representations via a fair filter (FairFil) network. To learn the FairFil, we introduce a contrastive learning framework that not only minimizes the correlation between filtered embeddings and bias words but also preserves rich semantic information of the original sentences. On real-world datasets, our FairFil effectively reduces the bias degree of pretrained text encoders, while continuously showing desirable performance on downstream tasks. Moreover, our post-hoc method does not require any retraining of the text encoders, further enlarging FairFil's application space.

arXiv commentAccepted by the 9th International Conference on Learning Representations (ICLR 2021)

References in corpus (9)

• Distributed Representations of Words and Phrases and their Compositionality
• BERTScore: Evaluating Text Generation with BERT
• Learning deep representations by mutual information estimation and maximization
• A Structured Self-attentive Sentence Embedding
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• Understanding the Origins of Bias in Word Embeddings
• Spatiotemporal Contrastive Video Representation Learning
• CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
• Learning Controllable Fair Representations

Cited by in corpus (7)

• KLUE: Korean Language Understanding Evaluation
• Learning Debiased Representation via Disentangled Feature Augmentation
• Fairness via Representation Neutralization
• Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools
• Modeling Techniques for Machine Learning Fairness: A Survey
• GroupMixNorm Layer for Learning Fair Models
• Debiasing Sentence Embedders through Contrastive Word Pairs