8 papers
Mind the Heads: Topological Representation Alignment for Multimodal LLMs
Davide Caffagni, Alberto Compagnoni, Federico Melis +5
Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an…
ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
Alberto Compagnoni, Marco Morini, Sara Sarto +5
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in jointly understanding text, images, and videos, often evaluated via Visual Question Answering (VQA).…
Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
Davide Caffagni, Sara Sarto, Marcella Cornia +5
Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in connecting vision and language, yet their proficiency in fundamental visual reasoning…
Recurrence Meets Transformers for Universal Multimodal Retrieval
Davide Caffagni, Sara Sarto, Marcella Cornia +2
With the rapid advancement of multimodal retrieval and its application in LLMs and multimodal LLMs, increasingly complex retrieval tasks have emerged. Existing methods predominantl…
Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
Alberto Compagnoni, Davide Caffagni, Nicholas Moratelli +3
Multimodal Large Language Models (MLLMs) emerge as a unified interface to address a multitude of tasks, ranging from NLP to computer vision. Despite showcasing state-of-the-art res…
LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
Federico Cocchi, Nicholas Moratelli, Davide Caffagni +4
Recent progress in Multimodal Large Language Models (MLLMs) has highlighted the critical roles of both the visual backbone and the underlying language model. While prior work has p…