8 papers
Distilling LLM Reasoning into Graph of Concept Predictors
Ziyang Yu, Liang Zhao
Deploying Large Language Models (LLMs) for discriminative workloads is often limited by inference latency, compute, and API costs at scale. Active distillation reduces these costs…
HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
Yizhao Gao, Jianyu Wei, Qihao Zhang +11
This work introduces Hybrid Sparse Attention (HySparse), a new architecture that interleaves each full attention layer with several sparse attention layers. While conceptually simp…
MiMo-Audio: Audio Language Models are Few-Shot Learners
Core Team, Dong Zhang, Gang Wang +97
Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks. In contrast, humans are able to generalize to new audio tasks with…
ToolCritic: Detecting and Correcting Tool-Use Errors in Dialogue Systems
Hassan Hamad, Yingru Xu, Liang Zhao +2
Tool-augmented large language models (LLMs) are increasingly employed in real-world applications, but tool usage errors still hinder their reliability. We introduce ToolCritic, a d…
CLaw: Benchmarking Chinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysis
Xinzhe Xu, Liang Zhao, Hongshen Xu +1
Large Language Models (LLMs) are increasingly tasked with analyzing legal texts and citing relevant statutes, yet their reliability is often compromised by general pre-training tha…
MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining
LLM-Core Xiaomi, :, Bingquan Xia +62
We present MiMo-7B, a large language model born for reasoning tasks, with optimization across both pre-training and post-training stages. During pre-training, we enhance the data p…