8 papers
A Multi-View Media Profiling Suite: Resources, Evaluation, and Analysis
Muhammad Arslan Manzoor, Dilshod Azizov, Daniil Orel +4
News outlets shape public opinion at a scale that makes automated detection of political bias and factuality essential. However, the field still lacks unified resources, comprehens…
Stress Testing Factual Consistency Metrics for Long-Document Summarization
Zain Muhammad Mujahid, Dustin Wright, Isabelle Augenstein
Evaluating the factual consistency of abstractive text summarization remains a significant challenge, particularly for long documents, where conventional metrics struggle with inpu…
LLM-as-a-qualitative-judge: automating error analysis in natural language generation
Nadezhda Chirkova, Tunde Oluwaseyi Ajayi, Seth Aycock +4
Prompting large language models (LLMs) to evaluate generated text, known as LLM-as-a-judge, has become a standard evaluation approach in natural language generation (NLG), but is p…
Unstructured Evidence Attribution for Long Context Query Focused Summarization
Dustin Wright, Zain Muhammad Mujahid, Lu Wang +2
Large language models (LLMs) are capable of generating coherent summaries from very long contexts given a user query, and extracting and citing evidence spans helps improve the tru…
Qorgau: Evaluating LLM Safety in Kazakh-Russian Bilingual Contexts
Maiya Goloburda, Nurkhan Laiyk, Diana Turmakhan +11
Large language models (LLMs) are known to have the potential to generate harmful content, posing risks to users. While significant progress has been made in developing taxonomies f…
Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts
Zain Muhammad Mujahid, Dilshod Azizov, Maha Tufail Agro +1
In an age characterized by the proliferation of mis- and disinformation online, it is critical to empower readers to understand the content they are reading. Important efforts in t…