Beyond Token Stacking: Multi-scale Visual Modulation and Adaptive Fused-feature Calibration for MLLMs
arXiv:2611.21168 · doi:10.1145/3767308.3836045
Abstract
Current Multimodal Large Language Models (MLLMs) have advanced rapidly and follow two main paradigms: one involves concatenating visual and textual tokens for input into Large Language Model (LLM), while the other injects visual knowledge into Feed-Forward Networks (FFNs) of the LLM. However, the former introduces a large number of redundant visual tokens, substantially increasing computational overhead. The latter reduces sequence length but removes images from the input stage entirely, which may hinder the activation of image-related knowledge. Moreover, naively injecting visual knowledge into the FFNs could induce activation distribution shifts due to cross-modal semantic discrepancies. To address these challenges, we develop Beyond Token Stacking (BTS), a novel visual enhancement framework for MLLMs. Specifically, we propose a Multi-scale Visual Modulation (MVM) mechanism that injects multi-scale visual information at the input stage via modulation without increasing the token sequence length. Furthermore, we develop an Adaptive Fused-feature Calibration (AFC) module that performs per-channel affine calibration to decouple visual injection from distribution alignment in the FFN layers, improving multimodal fusion stability while preserving language priors. Experiments show that our approach achieves state-of-the-art performance.
References in corpus (15)
- Learning Transferable Visual Models From Natural Language Supervision
- LLaMA: Open and Efficient Foundation Language Models
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- A Survey on Multimodal Large Language Models
- Learning multiple visual domains with residual adapters
- PaLI: A Jointly-Scaled Multilingual Language-Image Model
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
- GPT-4o System Card
- Parameter-Efficient Transfer Learning for NLP
- LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
- Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
- Multimodal Chain-of-Thought Reasoning in Language Models
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning