Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally

Arun Josephraj Arokiaraj, Zekun Wu, Adriano Koshiyama

A targeted adversarial perturbation can drive a vision-language model's (VLM's) teacher-forced training loss for a fixed target caption to near zero, yet the same model, allowed to…