Publications (18)
ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems
Qiaoling Chen, Zijun Liu, Peng Sun +6
Adapting large language models (LLMs) via reinforcement learning (RL) is often bottlenecked by the generation stage, which can consume over 75\% of the training time. Speculative d…
Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters
Zhengda Bian, Shenggui Li, Wei Wang +1
Efficient GPU resource scheduling is essential to maximize resource utilization and save training costs for the increasing amount of deep learning workloads in shared GPU clusters.…
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
Shenggui Li, Hongxin Liu, Zhengda Bian +5
The success of Transformer models has pushed the deep learning model scale to billions of parameters. Due to the limited memory resource of a single GPU, However, the best practice…
Sequence Parallelism: Long Sequence Training from System Perspective
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal +2
Transformer achieves promising results on various tasks. However, self-attention suffers from quadratic memory requirements with respect to the sequence length. Existing work focus…
TetriServe: Efficiently Serving Mixed DiT Workloads
Runyu Lu, Shiqi He, Wenxuan Tan +5
Diffusion Transformer (DiT) models excel at generating high-quality images through iterative denoising steps, but serving them under strict Service Level Objectives (SLOs) is chall…
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models
Yuliang Liu, Shenggui Li, Jiarui Fang +3
In recent years, large-scale models have demonstrated state-of-the-art performance across various domains. However, training such models requires various techniques to address the…
SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading
Qiaoling Chen, Shenggui Li, Wei Gao +3
In recent years, Large Language Models (LLMs) have exhibited remarkable capabilities, driving advancements in real-world applications. However, training LLMs on increasingly long i…
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
Cunxiao Du, Jing Jiang, Xu Yuanchen +8
Speculative decoding is a relatively new decoding framework that leverages small and efficient draft models to reduce the latency of LLMs. In this study, we introduce GliDe and CaP…
Sky Computing: Accelerating Geo-distributed Computing in Federated Learning
Jie Zhu, Shenggui Li, Yang You
Federated learning is proposed by Google to safeguard data privacy through training models locally on users' devices. However, with deep learning models growing in size to achieve…
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Jiangsu Du, Ziming Liu, Jiarui Fang +4
Large transformer models display promising performance on a wide range of natural language processing (NLP) tasks. Although the AI community has expanded the model scale to the tri…
CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
Qiaoling Chen, Zhisheng Ye, Tian Tang +7
Batch inference for agentic workloads stresses the GPU key-value (KV) cache in a sustained and cumulative manner, often causing severe throughput degradation well before memory cap…
Open-Sora: Democratizing Efficient Video Production for All
Zangwei Zheng, Xiangyu Peng, Tianji Yang +6
Vision and language are the two foundational senses for humans, and they build up our cognitive ability and intelligence. While significant breakthroughs have been made in AI langu…
Elixir: Train a Large Language Model on a Small GPU Cluster
Haichen Huang, Jiarui Fang, Hongxin Liu +2
In recent years, large language models have achieved great success due to their unprecedented size. However, training these models poses a challenge for most researchers as it requ…
An Efficient 2D Method for Training Super-Large Deep Learning Models
Qifan Xu, Shenggui Li, Chaoyu Gong +1
Huge neural network models have shown unprecedented performance in real-world applications. However, due to memory constraints, model parallelism must be utilized to host large mod…
SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Shenggui Li, Chao Wang, Yikai Zhu +14
Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to…
PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management
Jiarui Fang, Zilin Zhu, Shenggui Li +4
The pre-trained model (PTM) is revolutionizing Artificial Intelligence (AI) technology. However, the hardware requirement of PTM training is prohibitively high, making it a game fo…
A Frequency-aware Software Cache for Large Recommendation System Embeddings
Jiarui Fang, Geng Zhang, Jiatong Han +5
Deep learning recommendation models (DLRMs) have been widely applied in Internet companies. The embedding tables of DLRMs are too large to fit on GPU memory entirely. We propose a…
DSB: Dynamic Sliding Block Scheduling for Diffusion LLMs
Lizhuo Luo, Shenggui Li, Yonggang Wen +1
Diffusion large language models (dLLMs) have emerged as a promising alternative for text generation, distinguished by their native support for parallel decoding. In practice, block…