An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models

Haochen Luo, Jindong Gu, Fengyuan Liu +1

Different from traditional task-specific vision models, recent large VLMs can readily adapt to different vision tasks by simply using different textual instructions, i.e., prompts.…