7 papers
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
Shijie Cao, Qingyu Zhang, Boxi Yu +6
Omni modal large language models (OmniLLMs) have attracted wide attention for their ability to jointly process audio and video, but they generate large token sequences under audio-…
HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
Yizhao Gao, Jianyu Wei, Qihao Zhang +11
This work introduces Hybrid Sparse Attention (HySparse), a new architecture that interleaves each full attention layer with several sparse attention layers. While conceptually simp…
MiMo-V2-Flash Technical Report
Core Team, Bangjun Xiao, Bingquan Xia +123
We present MiMo-V2-Flash, a Mixture-of-Experts (MoE) model with 309B total parameters and 15B active parameters, designed for fast, strong reasoning and agentic capabilities. MiMo-…
MiMo-Audio: Audio Language Models are Few-Shot Learners
Core Team, Dong Zhang, Gang Wang +97
Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks. In contrast, humans are able to generalize to new audio tasks with…
Data Efficacy for Language Model Training
Yalun Dai, Yangyu Huang, Xin Zhang +6
Data is fundamental to the training of language models (LM). Recent research has been dedicated to data efficiency, which aims to maximize performance by selecting a minimal or opt…
SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
Yizhao Gao, Shuming Guo, Shijie Cao +12
We introduce SeerAttention-R, a sparse attention framework specifically tailored for the long decoding of reasoning models. Extended from SeerAttention, SeerAttention-R retains the…