6 papers
Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
Ruixuan Huang, Yipei Wang, Wenyi Fang +7
Frontier large language model training consumes massive accelerator fleets and long wall-clock computation, making stability failures costly when they occur. After a numerical or a…
Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs
Ruixuan Huang, Jinyuan Shi, Hantao Huang +5
We study dense-to-sparse continual training as a way to construct channel-sparse large language models from dense checkpoints. Starting from a Qwen2.5-8B dense backbone, we continu…
Casting a SPELL: Sentence Pairing Exploration for LLM Limitation-breaking
Yifan Huang, Xiaojun Jia, Wenbo Guo +4
Large language models (LLMs) have revolutionized software development through AI-assisted coding tools, enabling developers with limited programming expertise to create sophisticat…
AsyncHZP: Hierarchical ZeRO Parallelism with Asynchronous Scheduling for Scalable LLM Training
Huawei Bai, Yifan Huang, Wenqi Shi +4
The training efficiency and scalability of language models on massive clusters currently remain a critical bottleneck. Mainstream approaches like ND parallelism are often cumbersom…
Seed-Coder: Let the Code Model Curate Data for Itself
ByteDance Seed, Yuyu Zhang, Jing Su +24
Code data in large language model (LLM) pretraining is recognized crucial not only for code-related tasks but also for enhancing general intelligence of LLMs. Current open-source L…
A Vulnerability Code Intent Summary Dataset
Yifan Huang, Weisong Sun, Yubin Qu
In the era of Large Language Models (LLMs), the code summarization technique boosts a lot, along with the emergence of many new significant works. However, the potential of code su…