3 papers
cs.CV2025
MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
Dianyi Wang, Siyuan Wang, Zejun Li +6
Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across multi-modal tasks by scaling model size and training data. However, these dense LVLMs incur sig…
cs.LG2025
Mixture of Lookup Experts
Shibo Jie, Yehui Tang, Kai Han +4
Mixture-of-Experts (MoE) activates only a subset of experts during inference, allowing the model to maintain low inference FLOPs and latency even as the parameter count scales up.…
cs.CL2025
Unshackling Context Length: An Efficient Selective Attention Approach through Query-Key Compression
Haoyu Wang, Tong Teng, Tianyu Guo +4
Handling long-context sequences efficiently remains a significant challenge in large language models (LLMs). Existing methods for token selection in sequence extrapolation either e…