148 citations · 533 across the 14 of their papers we have counts for
8 papers · 1 filter
ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale
William Won, Taekyung Heo, Saeed Rashidi +3
As deep learning models and input data are scaling at an unprecedented rate, it is inevitable to move towards distributed training platforms to fit the model and increase training…
Mystique: Enabling Accurate and Scalable Generation of Production AI Benchmarks
Mingyu Liang, Wenyin Fu, Louis Feng +5
Building large AI fleets to support the rapidly growing DL workloads is an active research topic for modern cloud providers. Generating accurate benchmarks plays an essential role…
Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models
Saeed Rashidi, William Won, Sudarshan Srinivasan +2
Distributed training is a solution to reduce DNN training time by splitting the task across multiple NPUs (e.g., GPU/TPU). However, distributed training adds communication overhead…
Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models
Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang +50
Deep learning recommendation models (DLRMs) are used across many business-critical services at Facebook and are the single largest AI application in terms of infrastructure demand…
Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
Maxim Naumov, John Kim, Dheevatsa Mudigere +12
Large-scale training is important to ensure high performance and accuracy of machine-learning models. At Facebook we use many different models, including computer vision, video and…
Automatic Model Parallelism for Deep Neural Networks with Compiler and Hardware Support
Sanket Tavarageri, Srinivas Sridharan, Bharat Kaul
The deep neural networks (DNNs) have been enormously successful in tasks that were hitherto in the human-only realm such as image recognition, and language translation. Owing to th…
On Scale-out Deep Learning Training for Cloud and HPC
Srinivas Sridharan, Karthikeyan Vaidyanathan, Dhiraj Kalamkar +8
The exponential growth in use of large deep neural networks has accelerated the need for training these deep neural networks in hours or even minutes. This can only be achieved thr…
Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
Dipankar Das, Sasikanth Avancha, Dheevatsa Mudigere +5
We design and implement a distributed multinode synchronous SGD algorithm, without altering hyper parameters, or compressing data, or altering algorithmic behavior. We perform a de…