5 papers
Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitration
Yu Zhang, Mufan Xu, Xuefeng Bai +4
Modality following is the ability to selectively leverage multimodal contexts based on user instructions. It is fundamental to the safety and reliability of multimodal large langua…
Mitigating Multimodal Hallucination via Phase-wise Self-reward
Yu Zhang, Chuyang Sun, Kehai Chen +3
Large Vision-Language Models (LVLMs) still struggle with vision hallucination, where generated responses are inconsistent with the visual input. Existing methods either rely on lar…
Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition
Jinlong Ma, Yu Zhang, Xuefeng Bai +5
Grounded Multimodal Named Entity Recognition (GMNER) aims to extract text-based entities, assign them semantic categories, and ground them to corresponding visual regions. In this…
Evaluating and Steering Modality Preferences in Multimodal Large Language Model
Yu Zhang, Jinlong Ma, Yongshuai Hou +5
Multi-modal large language models (MLLMs) have achieved remarkable success on complex multi-modal tasks. However, it remains insufficiently explored whether they exhibit $\textbf{m…
BANER: Boundary-Aware LLMs for Few-Shot Named Entity Recognition
Quanjiang Guo, Yihong Dong, Ling Tian +3
Despite the recent success of two-stage prototypical networks in few-shot named entity recognition (NER), challenges such as over/under-detected false spans in the span detection s…