Line of Sight: On Linear Representations in VLLMs

Achyuta Rajaram, Sarah Schwettmann, Jacob Andreas +1

Language models can be equipped with multimodal capabilities by fine-tuning on embeddings of visual inputs. But how do such multimodal models represent images in their hidden activ…