27 papers
Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations
Haoyang Li, Fangcheng Fu, Hao Ge +6
The Single-Program Multiple-Data (SPMD) paradigm provides a unified abstraction to annotate various parallel dimensions in distributed deep learning (DL) training. With SPMD, users…
Adaptive Resource Management and Quality Control for Streaming Video Generation
Yifei Xia, Hao Yuan, Suhan Ling +5
Autoregressive diffusion transformers (AR-DiTs) recast video generation from an offline paradigm to a real-time streaming one: the model generates video one chunk at a time, making…
Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training
Yuming Zhou, Haoyang Li, Sheng Lin +6
Mixture-of-Experts (MoE) and reinforcement learning (RL) post-training now dominate large language model (LLM) development, yet expert load imbalance remains a critical challenge.…
DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
Yujie Wang, Siwei Chen, Longzan Luo +4
Reinforcement Learning (RL) has become pivotal for improving model capabilities yet suffers from rollout efficiency bottlenecks due to the long-tail response length distribution. W…
LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
Yifan Dai, Zhenhua Wu, Bohan Zeng +18
Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evid…
HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware
Ran Yan, Youhe Jiang, Xiaonan Nie +3
Training large language models (LLMs) is a computationally intensive task, which is typically conducted in data centers with homogeneous high-performance GPUs. In this paper, we ex…