Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Models

Abstract

Visual language pre-training (VLP) models have demonstrated significant success across various domains, yet they remain vulnerable to adversarial attacks. Addressing these adversarial vulnerabilities is crucial for enhancing security in multimodal learning. Traditionally, adversarial methods targeting VLP models involve simultaneously perturbing images and text. However, this approach faces notable challenges: first, adversarial perturbations often fail to translate effectively into real-world scenarios; second, direct modifications to the text are conspicuously visible. To overcome these limitations, we propose a novel strategy that exclusively employs image patches for attacks, thus preserving the integrity of the original text. Our method leverages prior knowledge from diffusion models to enhance the authenticity and naturalness of the perturbations. Moreover, to optimize patch placement and improve the efficacy of our attacks, we utilize the cross-attention mechanism, which encapsulates intermodal interactions by generating attention maps to guide strategic patch placements. Comprehensive experiments conducted in a white-box setting for image-to-text scenarios reveal that our proposed method significantly outperforms existing techniques, achieving a 100% attack success rate. Additionally, it demonstrates commendable performance in transfer tasks involving text-to-image configurations.

accepted by Visual Intelligence

References in corpus (10)

• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• Structure-measure: A New Way to Evaluate Foreground Maps
• JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection
• How Good is Google Bard's Visual Understanding? An Empirical Study on Open Challenges
• SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation
• AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models
• MOSE: A New Dataset for Video Object Segmentation in Complex Scenes
• Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval
• MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions

Cited by in corpus (3)

• A Review on Geometry and Surface Inspection in 3D Concrete Printing
• An Expert Ensemble for Detecting Anomalous Scenes, Interactions, and Behaviors in Autonomous Driving
• MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall