23 papers
Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model
Sibo Wang, Jie Zhang, Shiguang Shan +2
While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks…
Neural Gate: Mitigating Privacy Risks in LVLMs via Neuron-Level Gradient Gating
Xiangkui Cao, Jie Zhang, Meina Kan +2
Large Vision-Language Models (LVLMs) have shown remarkable potential across a wide array of vision-language tasks, leading to their adoption in critical domains such as finance and…
Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP
Sen Nie, Jie Zhang, Zhuo Wang +2
Vision-language models (VLMs) such as CLIP have demonstrated remarkable zero-shot generalization, yet remain highly vulnerable to adversarial examples (AEs). While test-time defens…
What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
Sen Nie, Jie Zhang, Zhongqi Wang +3
Achieving adversarial robustness in Vision-Language Models (VLMs) inevitably compromises accuracy on clean data, presenting a long-standing and challenging trade-off. In this work,…
MM-MoralBench: A MultiModal Moral Evaluation Benchmark for Large Vision-Language Models
Bei Yan, Jie Zhang, Zhiyuan Chen +2
The rapid integration of Large Vision-Language Models (LVLMs) into critical domains necessitates comprehensive moral evaluation to ensure their alignment with human values. While e…
VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
Sibo Wang, Xiangkui Cao, Jie Zhang +4
The emergence of Large Vision-Language Models (LVLMs) marks significant strides towards achieving general artificial intelligence. However, these advancements are accompanied by co…