70 citations · 205 across the 31 of their papers we have counts for
15 papers · 1 filter
Accelerating MoE Training through Expert Placement and Replica Coordination
Zhiqiang Tan, Maoxin Wang, Wenxiang Lin +4
Distributed training of sparsely activated mixture-of-experts (MoE) models is often bottlenecked by workload imbalance and all-to-all (A2A) communication induced by dynamic token r…
Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Training
Zhiqiang Tan, Maoxin Wang, Sijie Wang +4
It is well established that the reasoning capabilities of large language models (LLMs) can be improved by applying reinforcement learning (RL) in a post-training stage. In a standa…
Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis
Weile Luo, Ruibo Fan, Zeyu Li +4
This study presents a comprehensive multi-level analysis of the NVIDIA Hopper GPU architecture, focusing on its performance characteristics and novel features. We benchmark Hopper'…
FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
Zhenheng Tang, Xueze Kang, Yiming Yin +11
To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system desig…
BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
Yuxin Wang, Yuhan Chen, Zeyu Li +11
Serving systems for Large Language Models (LLMs) are often optimized to improve quality of service (QoS) and throughput. However, due to the lack of open-source LLM serving workloa…
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs
Zhenheng Tang, Yuxin Wang, Xin He +8
The rapid growth of memory and computation requirements of large language models (LLMs) has outpaced the development of hardware, hindering people who lack large-scale high-end GPU…
Energy-aware Task Scheduling with Deadline Constraint in DVFS-enabled Heterogeneous Clusters
Xinxin Mei, Qiang Wang, Xiaowen Chu +3
Energy conservation of large data centers for high-performance computing workloads, such as deep learning with big data, is of critical significance, where cutting down a few perce…
Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format
Shaohuai Shi, Qiang Wang, Xiaowen Chu
Multiplication of a sparse matrix to a dense matrix (SpDM) is widely used in many areas like scientific computing and machine learning. However, existing works under-look the perfo…
Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
Qiang Wang, Shaohuai Shi, Canhui Wang +1
Distributed Deep Learning (DDL) has rapidly grown its popularity since it helps boost the training performance on high-performance GPU clusters. Efficient job scheduling is indispe…
Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
Yuxin Wang, Qiang Wang, Shaohuai Shi +4
Deep learning has become widely used in complex AI applications. Yet, training a deep neural network (DNNs) model requires a considerable amount of calculations, long running time,…
A Distributed Synchronous SGD Algorithm with Global Top- Sparsification for Low Bandwidth Networks
Shaohuai Shi, Qiang Wang, Kaiyong Zhao +4
Distributed synchronous stochastic gradient descent (S-SGD) has been widely used in training large-scale deep neural networks (DNNs), but it typically requires very high communicat…
A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
Shaohuai Shi, Qiang Wang, Xiaowen Chu +1
With huge amounts of training data, deep learning has made great breakthroughs in many artificial intelligence (AI) applications. However, such large-scale data sets present comput…
Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
Shaohuai Shi, Qiang Wang, Xiaowen Chu
Deep learning frameworks have been widely deployed on GPU servers for deep learning applications in both academia and industry. In training deep neural networks (DNNs), there are m…
A Survey and Measurement Study of GPU DVFS on Energy Conservation
Xinxin Mei, Qiang Wang, Xiaowen Chu
Energy efficiency has become one of the top design criteria for current computing systems. The dynamic voltage and frequency scaling (DVFS) has been widely adopted by laptop comput…
Benchmarking State-of-the-Art Deep Learning Software Tools
Shaohuai Shi, Qiang Wang, Pengfei Xu +1
Deep learning has been shown as a successful machine learning method for a variety of tasks, and its popularity results in numerous open-source deep learning software tools. Traini…