2 citations · 5 across the 9 of their papers we have counts for
9 papers
HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing
Jianyu Wei, Yizhao Gao, Qihao Zhang +12
Long-horizon and multi-turn agents typically generate short actions and process long observations from tools and environments. This growing context demands efficient prefill, compa…
HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
Yizhao Gao, Jianyu Wei, Qihao Zhang +11
This work introduces Hybrid Sparse Attention (HySparse), a new architecture that interleaves each full attention layer with several sparse attention layers. While conceptually simp…
MiMo-V2-Flash Technical Report
Core Team, Bangjun Xiao, Bingquan Xia +123
We present MiMo-V2-Flash, a Mixture-of-Experts (MoE) model with 309B total parameters and 15B active parameters, designed for fast, strong reasoning and agentic capabilities. MiMo-…
MiMo-Audio: Audio Language Models are Few-Shot Learners
Core Team, Dong Zhang, Gang Wang +97
Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks. In contrast, humans are able to generalize to new audio tasks with…
iFairy: the First 2-bit Complex LLM with All Parameters in
Feiyu Wang, Guoan Wang, Yihao Zhang +7
Quantization-Aware Training (QAT) integrates quantization into the training loop, enabling LLMs to learn robust low-bit representations, and is widely recognized as one of the most…
MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining
LLM-Core Xiaomi, :, Bingquan Xia +62
We present MiMo-7B, a large language model born for reasoning tasks, with optimization across both pre-training and post-training stages. During pre-training, we enhance the data p…
MiMo-VL Technical Report
Core Team, Zihao Yue, Zhenru Lin +71
We open-source MiMo-VL-7B-SFT and MiMo-VL-7B-RL, two powerful vision-language models delivering state-of-the-art performance in both general visual understanding and multimodal rea…
INT-FlashAttention: Enabling Flash Attention for INT8 Quantization
Shimao Chen, Zirui Liu, Zhiying Wu +6
As the foundation of large language models (LLMs), self-attention module faces the challenge of quadratic time and memory complexity with respect to sequence length. FlashAttention…
Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing
Peizhuang Cong, Aomufei Yuan, Shimao Chen +4
MoE facilitates the development of large models by making the computational complexity of the model no longer scale linearly with increasing parameters. The learning sparse gating…