99 citations · 141 across the 18 of their papers we have counts for
21 papers
PlaneZip: Lossless Bit-Plane Compression of Training Activations Offloaded over PCIe
Zhufan Gu, Qizhe Tang, Yuxin Wang +2
Training under a device-memory budget relies on tensor rematerialization: an evicted activation is either swapped to host memory or dropped and recomputed. Because GPU compute has…
Zellige: Moldable Sequence Placement for Mixed Image-Video DiT Training
Guangyu Xiang, Xueze Kang, Minwei Zhao +4
High-quality video generation requires training Diffusion Transformers (DiTs) jointly on image and video data, posing a mixed-length sequence training problem across GPUs. Existing…
Xema: Efficient Diffusion Serving through Fine-Grained Memory Management and Auto-Configuration
Xueze Kang, Guangyu Xiang, Suyi Li +4
Diffusion models are increasingly deployed as production visual-generation services, where serving high-resolution image and long video generation is often limited by GPU memory. P…
KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding
Guangyu Xiang, Xueze Kang, Lin Zhang +4
LLM serving is increasingly dominated by long and dynamic decode workloads from agents, reasoning models, and extended conversations. When bursty long-context demand exceeds deploy…
On the Spectral Flattening of Quantized Embeddings
Junlin Huang, Wenyi Fang, Zhenheng Tang +5
Training Large Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic…
Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
Xinglin Pan, Shaohuai Shi, Wenxiang Lin +4
The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert act…
ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training
Xueze Kang, Guangyu Xiang, Yuxin Wang +16
Large-scale LLM pretraining now runs across -- accelerators, making failures routine and elasticity mandatory. We posit that an elastic-native training system must join…
DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization
Zhenheng Tang, Zichen Tang, Junlin Huang +7
The growth of large language models (LLMs) increases challenges of accelerating distributed training across multiple GPUs in different data centers. Moreover, concerns about data p…
FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
Zhenheng Tang, Xueze Kang, Yiming Yin +11
To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system desig…
ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
Xin He, Shunkang Zhang, Kaijie Tang +8
Sparse Mixture-of-Experts (MoE) models can outperform dense large language models at similar computation by activating only a small set of experts per token. However, stacking many…
Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
Zichen Tang, Junlin Huang, Rudan Yan +5
Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL). However, thes…
Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointing
Yuxin Wang, Xueze Kang, Shaohuai Shi +8
To efficiently scale large model (LM) training, researchers transition from data parallelism (DP) to hybrid parallelism (HP) on GPU clusters, which frequently experience hardware a…
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs
Zhenheng Tang, Yuxin Wang, Xin He +8
The rapid growth of memory and computation requirements of large language models (LLMs) has outpaced the development of hardware, hindering people who lack large-scale high-end GPU…
FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training
Zhenheng Tang, Xiaowen Chu, Ryan Yide Ran +7
Federated Learning (FL) enables collaborations among clients for train machine learning models while protecting their data privacy. Existing FL simulation platforms that are design…
NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension
Xin He, Jiangchao Yao, Yuxin Wang +5
One-shot neural architecture search (NAS) substantially improves the search efficiency by training one supernet to estimate the performance of every possible child architecture (i.…
A Survey of Transformers
Tianyang Lin, Yuxin Wang, Xiangyang Liu +1
Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natura…
A Survey of Deep Learning Techniques for Neural Machine Translation
Shuoheng Yang, Yuxin Wang, Xiaowen Chu
In recent years, natural language processing (NLP) has got great development with deep learning techniques. In the sub-field of machine translation, a new approach named Neural Mac…
Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models
Xin He, Shihao Wang, Shaohuai Shi +10
Skin disease is one of the most common types of human diseases, which may happen to everyone regardless of age, gender or race. Due to the high visual diversity, human diagnosis hi…
Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
Yuxin Wang, Qiang Wang, Shaohuai Shi +4
Deep learning has become widely used in complex AI applications. Yet, training a deep neural network (DNNs) model requires a considerable amount of calculations, long running time,…
The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study
Zhenheng Tang, Yuxin Wang, Qiang Wang +1
Over the past years, great progress has been made in improving the computing power of general-purpose graphics processing units (GPGPUs), which facilitates the prosperity of deep n…
A Distributed Synchronous SGD Algorithm with Global Top- Sparsification for Low Bandwidth Networks
Shaohuai Shi, Qiang Wang, Kaiyong Zhao +4
Distributed synchronous stochastic gradient descent (S-SGD) has been widely used in training large-scale deep neural networks (DNNs), but it typically requires very high communicat…