Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

Chenhong He, Lei Li, Shicheng Li +5

Hybrid attention dominates frontier LLMs, yet Vision Transformers (ViTs) in multimodal LLMs lack a satisfactory hybrid design, with no consensus on why certain attention patterns w…