8 papers
Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
Zhiwei Zhang, Zechen Sun, Fei Zhao +6
On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supe…
D-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Zechen Sun, Zhiwei Zhang, Fei Zhao +7
Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollou…
One Token Is Enough: Improving Diffusion Language Models with a Sink Token
Zihou Zhang, Zheyong Xie, Li Zhong +3
Diffusion Language Models (DLMs) have emerged as a compelling alternative to autoregressive approaches, enabling parallel text generation with competitive performance. Despite thes…
Balancing Understanding and Generation in Discrete Diffusion Models
Yue Liu, Yuzhong Zhao, Zheyong Xie +5
In discrete generative modeling, two dominant paradigms demonstrate divergent capabilities: Masked Diffusion Language Models (MDLM) excel at semantic understanding and zero-shot ge…
Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training
Shengrui Li, Fei Zhao, Kaiyan Zhao +6
Determining an effective data mixture is a key factor in Large Language Model (LLM) pre-training, where models must balance general competence with proficiency on hard tasks such a…
EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce
Kaiyan Zhao, Zijie Meng, Zheyong Xie +4
Large Language Model (LLM)-based agents are increasingly deployed in e-commerce applications to assist customer services in tasks such as product inquiries, recommendations, and or…
Benchmarking Machine Translation on Chinese Social Media Texts
Kaiyan Zhao, Zheyong Xie, Zhongtao Miao +3
The prevalence of rapidly evolving slang, neologisms, and highly stylized expressions in informal user-generated text, particularly on Chinese social media, poses significant chall…
RedOne 2.0: Rethinking Domain-specific LLM Post-Training in Social Networking Services
Fei Zhao, Chonggang Lu, Haofu Qian +9
As a key medium for human interaction and information exchange, social networking services (SNS) pose unique challenges for large language models (LLMs): heterogeneous workloads, f…