11 papers
MIC: Explaining Image-Claim Inconsistencies in AI-Generated Multimodal Misinformation
Ruihong Zeng, Jonathan Tonglet, Preslav Nakov +1
Claims paired with AI-generated images are a rapidly growing form of misinformation. Existing automated fact-checking (AFC) methods mainly treat this as a provenance problem, detec…
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
NewsRECON: News Article Retrieval for Image Contextualization
Jonathan Tonglet, Iryna Gurevych, Tinne Tuytelaars +1
Identifying when and where a news image was taken is crucial for journalists and forensic experts to produce credible stories and debunk misinformation. While many existing methods…
ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation
Jesus-German Ortiz-Barajas, Jonathan Tonglet, Vivek Gupta +1
Multimodal large language models (MLLMs) are increasingly used to automate chart generation from data tables, improving efficiency but introducing new misuse risks. We present Char…
M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
Jiahui Geng, Jonathan Tonglet, Iryna Gurevych
Existing real-world datasets for multimodal fact-checking have multiple limitations: they contain few instances, cover only one or two languages, focus on a single task, or rely on…
Is this chart lying to me? Automating the detection of misleading visualizations
Jonathan Tonglet, Jan Zimny, Tinne Tuytelaars +1
Misleading visualizations are a potent driver of misinformation on social media and the web. By violating chart design principles, they distort data and lead readers to draw inaccu…
COVE: COntext and VEracity prediction for out-of-context images
Jonathan Tonglet, Gabriel Thiem, Iryna Gurevych
Images taken out of their context are the most prevalent form of multimodal misinformation. Debunking them requires (1) providing the true context of the image and (2) checking the…
Protecting multimodal large language models against misleading visualizations
Jonathan Tonglet, Tinne Tuytelaars, Marie-Francine Moens +1
Visualizations play a pivotal role in daily communication in an increasingly data-driven world. Research on multimodal large language models (MLLMs) for automated chart understandi…
Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Framework
Zirui Song, Jingpu Yang, Yuan Huang +6
Geolocation, the task of identifying an image's location, requires complex reasoning and is crucial for navigation, monitoring, and cultural preservation. However, current methods…
"Image, Tell me your story!" Predicting the original meta-context of visual misinformation
Jonathan Tonglet, Marie-Francine Moens, Iryna Gurevych
To assist human fact-checkers, researchers have developed automated approaches for visual misinformation detection. These methods assign veracity scores by identifying inconsistenc…
SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQA
Jonathan Tonglet, Manon Reusens, Philipp Borchert +1
Question answering over hybrid contexts is a complex task, which requires the combination of information extracted from unstructured texts and structured tables in various ways. Re…