6 papers
Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP
Naman Deep Singh, Francesco Croce, Matthias Hein
Vision-Language models like CLIP have been shown to be highly effective at linking visual perception and natural language understanding, enabling sophisticated image-text capabilit…
On the Adversarial Robustness of Discrete Image Tokenizers
Rishika Bhagwatkar, Irina Rish, Nicolas Flammarion +1
Discrete image tokenizers encode visual inputs as sequences of tokens from a finite vocabulary and are gaining popularity in multimodal systems, including encoder-only, encoder-dec…
On the Out-of-Distribution Generalization of Reasoning in Multimodal LLMs for Simple Visual Planning Tasks
Yannic Neuhaus, Nicolas Flammarion, Matthias Hein +1
Integrating reasoning in large language models and large vision-language models has recently led to significant improvement of their capabilities. However, the generalization of re…
Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
Naman Deep Singh, Maximilian Müller, Francesco Croce +1
Unlearning in large language models (LLMs) involves precisely removing specific information from a pre-trained model. This is crucial to ensure safety of LLMs by deleting private d…
FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens
Christian Schlarmann, Francesco Croce, Nicolas Flammarion +1
Contrastive language-image pre-training aligns features of text-image pairs in a common latent space via distinct encoders for each modality. While this approach achieves impressiv…
Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics
Francesco Croce, Christian Schlarmann, Naman Deep Singh +1
Measuring perceptual similarity is a key tool in computer vision. In recent years perceptual metrics based on features extracted from neural networks with large and diverse trainin…