598 citations · 1k across the 30 of their papers we have counts for
31 papers
On the Adversarial Robustness of Discrete Image Tokenizers
Rishika Bhagwatkar, Irina Rish, Nicolas Flammarion +1
Discrete image tokenizers encode visual inputs as sequences of tokens from a finite vocabulary and are gaining popularity in multimodal systems, including encoder-only, encoder-dec…
On the Out-of-Distribution Generalization of Reasoning in Multimodal LLMs for Simple Visual Planning Tasks
Yannic Neuhaus, Nicolas Flammarion, Matthias Hein +1
Integrating reasoning in large language models and large vision-language models has recently led to significant improvement of their capabilities. However, the generalization of re…
Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
Naman Deep Singh, Maximilian Müller, Francesco Croce +1
Unlearning in large language models (LLMs) involves precisely removing specific information from a pre-trained model. This is crucial to ensure safety of LLMs by deleting private d…
FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens
Christian Schlarmann, Francesco Croce, Nicolas Flammarion +1
Contrastive language-image pre-training aligns features of text-image pairs in a common latent space via distinct encoders for each modality. While this approach achieves impressiv…
Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics
Francesco Croce, Christian Schlarmann, Naman Deep Singh +1
Measuring perceptual similarity is a key tool in computer vision. In recent years perceptual metrics based on features extracted from neural networks with large and diverse trainin…
Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP
Naman Deep Singh, Francesco Croce, Matthias Hein
Vision-Language models like CLIP have been shown to be highly effective at linking visual perception and natural language understanding, enabling sophisticated image-text capabilit…
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs
Yuval Reif, Roy Schwartz
Large language models (LLMs) have shown remarkable adaptability to diverse tasks, by leveraging context prompts containing instructions, or minimal input-output examples. However,…
Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
Christian Schlarmann, Naman Deep Singh, Francesco Croce +1
Multi-modal foundation models like OpenFlamingo, LLaVA, and GPT-4 are increasingly used for various real-world tasks. Prior work has shown that these models are highly vulnerable t…
Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning
Hao Zhao, Maksym Andriushchenko, Francesco Croce +1
There is a consensus that instruction fine-tuning of LLMs requires high-quality data, but what are they? LIMA (NeurIPS 2023) and AlpaGasus (ICLR 2024) are state-of-the-art methods…
Segment (Almost) Nothing: Prompt-Agnostic Adversarial Attacks on Segmentation Models
Francesco Croce, Matthias Hein
General purpose segmentation models are able to generate (semantic) segmentation masks from a variety of prompts, including visual (points, boxed, etc.) and textual (object names)…
Towards Reliable Evaluation and Fast Training of Robust Semantic Segmentation Models
Francesco Croce, Naman D Singh, Matthias Hein
Adversarial robustness has been studied extensively in image classification, especially for the -threat model, but significantly less so for related tasks such as obje…
Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models
Naman D Singh, Francesco Croce, Matthias Hein
While adversarial training has been extensively studied for ResNet architectures and low resolution datasets like CIFAR, much less is known for ImageNet. Given the recent debate ab…
Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shifts
Francesco Croce, Sylvestre-Alvise Rebuffi, Evan Shelhamer +1
Adversarial training is widely used to make classifiers robust to a specific threat or adversary, such as -norm bounded perturbations of a given -norm. However, existing…
A Modern Look at the Relationship between Sharpness and Generalization
Maksym Andriushchenko, Francesco Croce, Maximilian Müller +2
Sharpness of minima is a promising quantity that can correlate with generalization in deep networks and, when optimized during training, can improve generalization. However, standa…
Diffusion Visual Counterfactual Explanations
Maximilian Augustin, Valentyn Boreiko, Francesco Croce +1
Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image classifier. They are 'small' but 'realistic' semantic changes of the image c…
Revisiting adapters with adversarial training
Sylvestre-Alvise Rebuffi, Francesco Croce, Sven Gowal
While adversarial training is generally used as a defense mechanism, recent works show that it can also act as a regularizer. By co-training a neural network on clean and adversari…
On the interplay of adversarial robustness and architecture components: patches, convolution and attention
Francesco Croce, Matthias Hein
In recent years novel architecture components for image classification have been developed, starting with attention and patches used in transformers. While prior works have analyze…
Sparse Visual Counterfactual Explanations in Image Space
Valentyn Boreiko, Maximilian Augustin, Francesco Croce +2
Visual counterfactual explanations (VCEs) in image space are an important tool to understand decisions of image classifiers as they show under which changes of the image the decisi…
Evaluating the Adversarial Robustness of Adaptive Test-time Defenses
Francesco Croce, Sven Gowal, Thomas Brunner +3
Adaptive defenses, which optimize at test time, promise to improve adversarial robustness. We categorize such adaptive test-time defenses, explain their potential benefits and draw…
Adversarial Robustness against Multiple and Single -Threat Models via Quick Fine-Tuning of Robust Classifiers
Francesco Croce, Matthias Hein
A major drawback of adversarially robust models, in particular for large scale datasets like ImageNet, is the extremely long training time compared to standard ones. Moreover, mode…
Mind the box: -APGD for sparse adversarial attacks on image classifiers
Francesco Croce, Matthias Hein
We show that when taking into account also the image domain , established -projected gradient descent (PGD) attacks are suboptimal as they do not consider that the ef…
RobustBench: a standardized adversarial robustness benchmark
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag +5
As a research community, we are still lacking a systematic understanding of the progress on adversarial robustness which often makes it hard to identify the most promising ideas in…
Sparse-RS: a versatile framework for query-efficient sparse black-box adversarial attacks
Francesco Croce, Maksym Andriushchenko, Naman D. Singh +2
We propose a versatile framework based on random search, Sparse-RS, for score-based sparse targeted and untargeted attacks in the black-box setting. Sparse-RS does not rely on subs…
Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
Francesco Croce, Matthias Hein
The field of defense strategies against adversarial attacks has significantly grown over the last years, but progress is hampered as the evaluation of adversarial defenses is often…