5 papers
Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models
Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza +5
Vision-language models (VLMs) are fragile under image corruption. We find that the wording of the question affects VLMs in two opposite ways. Verbose questions make VLMs substantia…
A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models
Mujtaba Hussain Mirza, Antonio D'Orazio, Odelia Melamed +1
Despite the rapid progress in multimodal models and Large Visual-Language Models (LVLM), they remain highly susceptible to adversarial perturbations, raising serious concerns about…
Understanding Adversarial Training with Energy-based Models
Mujtaba Hussain Mirza, Maria Rosaria Briglia, Filippo Bartolucci +3
We aim at using Energy-based Model (EBM) framework to better understand adversarial training (AT) in classifiers, and additionally to analyze the intrinsic generative capabilities…
What is Adversarial Training for Diffusion Models?
Briglia Maria Rosaria, Mujtaba Hussain Mirza, Giuseppe Lisanti +1
We answer the question in the title, showing that adversarial training (AT) for diffusion models (DMs) fundamentally differs from classifiers: while AT in classifiers enforces outp…
Shedding More Light on Robust Classifiers under the lens of Energy-based Models
Mujtaba Hussain Mirza, Maria Rosaria Briglia, Senad Beadini +1
By reinterpreting a robust discriminative classifier as Energy-based Model (EBM), we offer a new take on the dynamics of adversarial training (AT). Our analysis of the energy lands…