10 papers
AutoSFT: An Agentic System for Automated Data Research in Supervised Fine-Tuning
Bingxiang He, Pengqi Sun, Shihao Peng +18
Language model agents increasingly automate post-training, but when given the full training stack they converge on supervised fine-tuning and spend most effort revising training da…
UltraData-Code: From Raw Repositories to Algorithmically Dense and Task-Oriented Code Data
Chengying Tu, Hengyu Zhao, Shuaikang Xue +13
Code generation has become a core capability of large language models (LLMs), and code data is central to the pre-training process that develops it. As obtaining public code of suf…
ForgeMegakernel: A General Framework for Efficient Auto-Regressive Model Decode Megakernels
Leshan Li, Zhui Zhu, Xianglong Deng +6
Auto-regressive model decode is bandwidth-bound, since every weight and key/value-cache byte crosses high-bandwidth memory once per token. A megakernel is an ideal solution, but ex…
Rethinking On-Policy Distillation of Large Language Models II: One Training Example
Zixuan Fu, Bingxiang He, Yuxin Zuo +10
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leavi…
UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing
Xinlong Zhao, Dongsheng Liu, Hengyu Zhao +9
As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish. Consequently, improving Large Language Models (LLMs) now depends less on d…
UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models
Qundong Shi, Jie Zhou, Biyuan Lin +8
The development of audio foundation models has accelerated rapidly since the emergence of GPT-4o. However, the lack of comprehensive evaluation has become a critical bottleneck for…
VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning
Yixuan Zhou, Guoyang Zeng, Xin Liu +9
Generative models for speech synthesis face a fundamental trade-off: discrete tokens ensure stability but sacrifice expressivity, while continuous signals retain acoustic richness…
Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data
Yudong Wang, Zixuan Fu, Jie Cai +9
Data quality has become a key factor in enhancing model performance with the rapid development of large language models (LLMs). Model-driven data filtering has increasingly become…
DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Models
Ranchi Zhao, Zhen Leng Thai, Yifan Zhang +6
The performance of Large Language Models (LLMs) is substantially influenced by the pretraining corpus, which consists of vast quantities of unsupervised data processed by the model…
PyBench: Evaluating LLM Agent on various real-world coding tasks
Yaolun Zhang, Yinxu Pan, Yudong Wang +1
The LLM Agent, equipped with a code interpreter, is capable of automatically solving real-world coding tasks, such as data analysis and image editing. However, existing benchmarks…