Towards an Enhanced Understanding of Bias in Pre-trained Neural Language Models: A Survey with Special Emphasis on Affective Bias

Abstract

The remarkable progress in Natural Language Processing (NLP) brought about by deep learning, particularly with the recent advent of large pre-trained neural language models, is brought into scrutiny as several studies began to discuss and report potential biases in NLP applications. Bias in NLP is found to originate from latent historical biases encoded by humans into textual data which gets perpetuated or even amplified by NLP algorithm. We present a survey to comprehend bias in large pre-trained language models, analyze the stages at which they occur in these models, and various ways in which these biases could be quantified and mitigated. Considering wide applicability of textual affective computing based downstream tasks in real-world systems such as business, healthcare, education, etc., we give a special emphasis on investigating bias in the context of affect (emotion) i.e., Affective Bias, in large pre-trained language models. We present a summary of various bias evaluation corpora that help to aid future research and discuss challenges in the research on bias in pre-trained language models. We believe that our attempt to draw a comprehensive view of bias in pre-trained language models, and especially the exploration of affective bias will be highly beneficial to researchers interested in this evolving field.

arXiv commentAccepted at ICDSE 2021

References in corpus (10)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Semantics derived automatically from language corpora contain human-like biases
• Pre-trained Models for Natural Language Processing: A Survey
• Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes
• Getting Gender Right in Neural Machine Translation
• Fairway: A Way to Build Fair ML Software
• Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases
• Sentiment Analysis of Financial News Articles using Performance Indicators
• Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types
• BiasRV: Uncovering Biased Sentiment Predictions at Runtime

Cited by in corpus (2)

• Towards Exploring Fairness in Visual Transformer based Natural and GAN Image Detection Systems
• REDAffectiveLM: Leveraging Affect Enriched Embedding and Transformer-based Neural Language Model for Readers' Emotion Detection