Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

Josh Barua, Kyle Mahowald, William Rudman

Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--b…