10 papers
Preference Redirection via Attention Concentration: An Attack on Computer Use Agents
Dominik Seip, Matthias Hein
Advancements in multimodal foundation models have enabled the development of Computer Use Agents (CUAs) capable of autonomously interacting with GUI environments. As CUAs are not r…
Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP
Naman Deep Singh, Francesco Croce, Matthias Hein
Vision-Language models like CLIP have been shown to be highly effective at linking visual perception and natural language understanding, enabling sophisticated image-text capabilit…
Visual Memory Injection Attacks for Multi-Turn Conversations
Christian Schlarmann, Matthias Hein
Generative large vision-language models (LVLMs) have recently achieved impressive performance gains, and their user base is growing rapidly. However, the security of LVLMs, in part…
Robustness in Both Domains: CLIP Needs a Robust Text Encoder
Elias Abad Rocamora, Christian Schlarmann, Naman Deep Singh +3
Adversarial input attacks can cause a significant shift of CLIP embeddings. This can affect the downstream robustness of models incorporating CLIP in the pipeline, such as text-to-…
Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
Naman Deep Singh, Maximilian Müller, Francesco Croce +1
Unlearning in large language models (LLMs) involves precisely removing specific information from a pre-trained model. This is crucial to ensure safety of LLMs by deleting private d…
FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens
Christian Schlarmann, Francesco Croce, Nicolas Flammarion +1
Contrastive language-image pre-training aligns features of text-image pairs in a common latent space via distinct encoders for each modality. While this approach achieves impressiv…