Rethinking Projector Training in Multimodal LLMs

Fabian Gröger, Naël Ouerghemi, Shuo Wen +1

Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of…