A Surprising Failure? Multimodal LLMs and the NLVR Challenge

Anne Wu, Kianté Brantley, Yoav Artzi

This study evaluates three state-of-the-art MLLMs -- GPT-4V, Gemini Pro, and the open-source model IDEFICS -- on the compositional natural language vision reasoning task NLVR. Give…