A parameter-efficient three-branch architecture for multimodal misinformation detection with limited annotations

Abstract

With the rise of easily accessible generative tools for creating and manipulating multimedia content, the threat of realistic synthetic alterations to digital media, often involving manipulations across multiple modalities simultaneously, has grown exponentially. Recently, such techniques have been increasingly employed to distort narratives of important events and to spread misinformation on social media, prompting the development of misinformation detectors. In the context of misinformation conveyed through image-text pairs, several detection methods have been proposed. However, these approaches typically rely on computationally intensive architectures that often require large amounts of annotated data. In this work we introduce LADLE-MM: Limited Annotation based Detector with Learned Ensembles for Multimodal Misinformation, a model-soup initialized multimodal misinformation detector designed to operate under limited annotation setup and constrained training resources. LADLE-MM is composed of two unimodal branches and a third multimodal one that enhances image and text representations with additional multimodal embeddings extracted from BLIP, serving as fixed reference space. Tested on the DGM4 benchmark, our approach obtains competitive performance on both binary and multi-label classification tasks. Despite using 60.3% fewer trainable parameters than previous state-of-the-art models, it outperforms existing methods when trained under a limited annotation setup, specifically without grounding annotations. Moreover, when evaluated on the VERITE dataset, LADLE-MM outperforms current state-of-the-art-approaches that utilize more complex architectures involving Large Vision-Language-Models, demonstrating the effective generalization ability in an open-set setting and strong robustness to unimodal bias.

18 pages, 6 figures. Code available at https://github.com/ALCOR-Lab-DIAG/LADLE-MM