23 citations · 58 across the 11 of their papers we have counts for
11 papers
MonkeyTree: Near-Minimal Congestion for Multi-tenant Training via Migration
Anton A. Zabreyko, Weiyang Wang, Manya Ghobadi
We present MonkeyTree, the first system to mitigate network congestion in multi-tenant GPU clusters through job-migration based defragmentation rather than network-layer techniques…
A High-Throughput GPU Framework for Adaptive Lossless Compression of Floating-Point Data
Zheng Li, Weiyan Wang, Ruiyuan Li +5
The torrential influx of floating-point data from domains like IoT and HPC necessitates high-performance lossless compression to mitigate storage costs while preserving absolute da…
Checkmate: Zero-Overhead Model Checkpointing via Network Gradient Replication
Ankit Bhardwaj, Weiyang Wang, Jeremy Carin +2
This paper presents Checkmate, a system that enables per-iteration checkpointing in DNN training without any training slowdown. The traditional approach to checkpointing requires a…
Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Services
Weiyan Wang, Yilun Jin, Yiming Zhang +7
Due to high accuracy, BERT-like models have been widely adopted by text mining and web searching. However, large BERT-like models suffer from inefficient online inference, facing t…
Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters
Weiyang Wang, Manya Ghobadi, Kayvon Shakeri +2
This paper presents a low-cost network architecture for training large language models (LLMs) at hyperscale. We study the optimal parallelization strategy of LLMs and propose a nov…
Efficient Direct-Connect Topologies for Collective Communications
Liangyu Zhao, Siddharth Pal, Tapan Chugh +5
We consider the problem of distilling efficient network topologies for collective communications. We provide an algorithmic framework for constructing direct-connect topologies opt…
TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs
Weiyang Wang, Moein Khazraee, Zhizhen Zhong +5
We propose TopoOpt, a novel direct-connect fabric for deep neural network (DNN) training workloads. TopoOpt co-optimizes the distributed training process across three dimensions: c…
Multi-Objective Congestion Control
Yiqing Ma, Han Tian, Xudong Liao +4
Decades of research on Internet congestion control (CC) has produced a plethora of algorithms that optimize for different performance objectives. Applications face the challenge of…
Domain-specific Communication Optimization for Distributed DNN Training
Hao Wang, Jingrong Chen, Xinchen Wan +7
Communication overhead poses an important obstacle to distributed DNN training and draws increasing attention in recent years. Despite continuous efforts, prior solutions such as g…
DS-Sync: Addressing Network Bottlenecks with Divide-and-Shuffle Synchronization for Distributed DNN Training
Weiyan Wang, Cengguang Zhang, Liu Yang +2
Bulk synchronous parallel (BSP) is the de-facto paradigm for distributed DNN training in today's production clusters. However, due to the global synchronization nature, its perform…
Quantifying the Performance of Federated Transfer Learning
Qinghe Jing, Weiyan Wang, Junxue Zhang +2
The scarcity of data and isolated data islands encourage different organizations to share data with each other to train machine learning models. However, there are increasing conce…