Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition

Peng Xie, Amr Alanwar

Language models compute over tokens: language is their input, their output, and increasingly their internal representation. Whether language should keep all of these positions depe…