Better Language Models Exhibit Higher Visual Alignment

Jona Ruthardt, Gertjan J. Burghouts, Serge Belongie +1

How well do text-only large language models (LLMs) align with the visual world? We present a systematic evaluation of this question by incorporating frozen representations of vario…