Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection

Abstract

Although media bias detection is a complex multi-task problem, there is, to date, no unified benchmark grouping these evaluation tasks. We introduce the Media Bias Identification Benchmark (MBIB), a comprehensive benchmark that groups different types of media bias (e.g., linguistic, cognitive, political) under a common framework to test how prospective detection techniques generalize. After reviewing 115 datasets, we select nine tasks and carefully propose 22 associated datasets for evaluating media bias detection techniques. We evaluate MBIB using state-of-the-art Transformer techniques (e.g., T5, BART). Our results suggest that while hate speech, racial bias, and gender bias are easier to detect, models struggle to handle certain bias types, e.g., cognitive and political bias. However, our results show that no single technique can outperform all the others significantly. We also find an uneven distribution of research interest and resource allocation to the individual tasks in media bias. A unified benchmark encourages the development of more robust systems and shifts the current paradigm in media bias detection evaluation towards solutions that tackle not one but multiple media bias types simultaneously.

arXiv commentTo be published in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23)

References in corpus (17)

• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model
• Neural Media Bias Detection Using Distant Supervision With BABE -- Bias Annotations By Experts
• Female Librarians and Male Computer Programmers? Gender Bias in Occupational Images on Digital Media Platforms
• Neural Based Statement Classification for Biased Language
• An Automated Pipeline for Character and Relationship Extraction from Readers' Literary Book Reviews on Goodreads.com
• Gender Classification and Bias Mitigation in Facial Images
• Exploiting Transformer-based Multitask Learning for the Detection of Media Bias in News Articles
• Us vs. Them: A Dataset of Populist Attitudes, News Bias and Emotions
• A Domain-adaptive Pre-training Approach for Language Bias Detection in News
• The Media Bias Taxonomy: A Systematic Literature Review on the Forms and Automated Detection of Media Bias
• Automatically Neutralizing Subjective Bias in Text
• In Plain Sight: Media Bias Through the Lens of Factual Reporting
• Mitigating Media Bias through Neutral Article Generation
• POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection
• Modeling Ideological Salience and Framing in Polarized Online Groups with Graph Neural Networks and Structured Sparsity

Cited by in corpus (4)

• ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models
• Improved Models for Media Bias Detection and Subcategorization
• Mapping the Media Landscape: Predicting Factual Reporting and Political Bias Through Web Interactions
• DocNet: Semantic Structure in Inductive Bias Detection Models