Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models

Tianyou Jiang

Multimodal Large Language Models (MLLMs) commonly reuse visual encoders pretrained with CLIP, although the features of these ViTs are ultimately consumed by autoregressive LLMs. We…