Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling

Junlin Li, Guodong DU, Jing Li +8

Fine-tuning Large Language Models (LLMs) with multimodal encoders on modality-specific data expands the modalities that LLMs can handle, leading to the formation of Multimodal LLMs…