Showing cs.DCShow all
3 papers · 1 filter
cs.DC2026
MemX: Memory-Aware Automatic Parallelism for Cross-Vendor Heterogeneous GPU Training
Lin Meng, Yuzhong Sun, Jie Zhu
Cross-vendor heterogeneous GPU clusters are increasingly used for large-scale model training. However, their GPUs differ substantially in compute power, memory capacity, and interc…
cs.DC2025
DeFT: Mitigating Data Dependencies for Flexible Communication Scheduling in Distributed Training
Lin Meng, Yuzhong Sun
Communication scheduling aims to reduce communication bottlenecks in data parallel training (DP) by maximizing the overlap between computation and communication. However, existing…
cs.DC2023
Near-Linear Scaling Data Parallel Training with Overlapping-Aware Gradient Compression
Lin Meng, Yuzhong Sun, Weimin Li
Existing Data Parallel (DP) trainings for deep neural networks (DNNs) often experience limited scalability in speedup due to substantial communication overheads. While Overlapping…