7 papers · 1 filter
Detecting Chain-of-Thought Hijacking in Backdoored Multimodal Large Reasoning Models
Zhifang Zhang, Qi Wei, Weitong Chen +5
Chain-of-thought (CoT) reasoning enables multimodal large reasoning models (MLRMs) to solve complex visual problems, but also exposes them to CoT hijacking: an adversary implants a…
Test-Time Attention Purification for Backdoored Large Vision Language Models
Zhifang Zhang, Bojun Yang, Shuo He +5
Despite the strong multimodal performance, large vision-language models (LVLMs) are vulnerable during fine-tuning to backdoor attacks, where adversaries insert trigger-embedded sam…
TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models
Zhifang Zhang, Qiqi Tao, Jiaqi Lv +3
Large vision-language models (LVLMs) have achieved impressive performance across a wide range of vision-language tasks, while they remain vulnerable to backdoor attacks. Existing b…
Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Models
Zhifang Zhang, Jiahan Zhang, Shengjie Zhou +4
Multimodal pre-trained models (e.g., ImageBind), which align distinct data modalities into a shared embedding space, have shown remarkable success across downstream tasks. However,…
LanP: Rethinking the Impact of Language Priors in Large Vision-Language Models
Zongyu Wu, Yuwei Niu, Hongcheng Gao +12
Large Vision-Language Models (LVLMs) have shown impressive performance in various tasks. However, LVLMs suffer from hallucination, which hinders their adoption in the real world. E…
Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning
Zhifang Zhang, Shuo He, Haobo Wang +2
Multimodal contrastive learning models (e.g., CLIP) can learn high-quality representations from large-scale image-text datasets, while they exhibit significant vulnerabilities to b…
Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
Zhifang Zhang, Yuwei Niu, Xin Liu +1
Vision-language models (VLMs) can learn high-quality representations from a large-scale training dataset of image-text pairs. Prompt learning is a popular approach to fine-tuning V…