8 papers
ForgeMegakernel: A General Framework for Efficient Auto-Regressive Model Decode Megakernels
Leshan Li, Zhui Zhu, Xianglong Deng +6
Auto-regressive model decode is bandwidth-bound, since every weight and key/value-cache byte crosses high-bandwidth memory once per token. A megakernel is an ideal solution, but ex…
Rethinking On-Policy Distillation of Large Language Models II: One Training Example
Zixuan Fu, Bingxiang He, Yuxin Zuo +10
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leavi…
UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing
Xinlong Zhao, Dongsheng Liu, Hengyu Zhao +9
As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish. Consequently, improving Large Language Models (LLMs) now depends less on d…
UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models
Qundong Shi, Jie Zhou, Biyuan Lin +8
The development of audio foundation models has accelerated rapidly since the emergence of GPT-4o. However, the lack of comprehensive evaluation has become a critical bottleneck for…
VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning
Yixuan Zhou, Guoyang Zeng, Xin Liu +9
Generative models for speech synthesis face a fundamental trade-off: discrete tokens ensure stability but sacrifice expressivity, while continuous signals retain acoustic richness…
Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data
Yudong Wang, Zixuan Fu, Jie Cai +9
Data quality has become a key factor in enhancing model performance with the rapid development of large language models (LLMs). Model-driven data filtering has increasingly become…