7 citations · 14 across the 6 of their papers we have counts for
7 papers
SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
Yuchen Wang, Qihui Zhu, Yang Liu +2
Recent multimodal large language models (MLLMs), such as Qwen2.5-VL and InternVL3, generate large numbers of vision tokens for high-resolution inputs, leading to substantial comput…
RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models
Qihui Zhu, Yuchen Wang, Zijian Wen +7
On-Policy Self-Distillation (OPSD) uses privileged information available only to the teacher to provide dense token-level supervision on trajectories generated by the student. Howe…
HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models
Qihui Zhu, Tao Zhang, Yuchen Wang +9
In multimodal large language models (MLLMs), the surge of visual tokens significantly increases the inference time and computational overhead, making them impractical for real-time…
Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
Zheng Zhan, Liliang Ren, Shuohang Wang +5
Linear State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant inference-time computation and memory complexity. Recent advances,…
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
Microsoft, :, Abdelrahman Abouelenin +73
We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language model trained on high-qualit…
Efficiently Editing Mixture-of-Experts Models with Compressed Experts
Yifei He, Yang Liu, Chen Liang +1
Mixture-of-Experts (MoE) models have become a key approach for scaling large language models efficiently by activating only a subset of experts during training and inference. Typic…
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
Liliang Ren, Yang Liu, Yadong Lu +3
Efficiently modeling sequences with infinite context length has long been a challenging problem. Previous approaches have either suffered from quadratic computational complexity or…