4 papers
AoiZora: Topology-Aware Auto-Parallel Optimization for Inference of Diffusion Transformers
Kaijian Wang, Yuanyuan Xu, Fanjiang Ye +5
Video diffusion has quickly grown into a key generative serving workload, yet producing each clip demands many denoising iterations over large spatio-temporal latents, which puts l…
Characterization-Guided GPU Fault Resilience in NVIDIA MPS
Rixin Liu, Xingqi Cui, Kaijian Wang +4
NVIDIA Multi-Process Service (MPS) enables fine-grained GPU sharing by allowing multiple processes to execute concurrently on the same GPU, making it an important mechanism for imp…
ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads
Jingwei Zuo, Xinze Feng, Zien Liu +5
Low-Rank Adaptation (LoRA) is now the dominant method for parameter-efficient fine-tuning of large language models, but achieving a high-quality adapter often requires systematic h…
Empowering Distributed Training with Sparsity-driven Data Synchronization
Zhuang Wang, Zhaozhuo Xu, Jingyi Xi +3
Distributed training is the de facto standard to scale up the training of deep learning models with multiple GPUs. Its performance bottleneck lies in communications for gradient sy…