78 citations · 160 across the 13 of their papers we have counts for
13 papers
NUNA: Characterizing and Mitigating Non-Uniform Network Access in Multi-Die GPU Scale-Up Systems
Conor James Green, William Won, Tuan Ta +1
Graphics processing unit (GPU) architectures are growing in size to meet the increasing compute and memory requirements. As GPU sizes increase, intra-socket wire transfer delay inc…
ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modeling
William Won, Jinsun Yoo, Tuan Ta +16
Distributed machine learning (ML) is a key paradigm for today's large-scale artificial intelligence applications. As model inference arises as an important use case, faithful model…
PCCL: Process Group-Aware Scalable and Generic Collective Algorithm Synthesizer
William Won, Kartik Lakhotia, Madhu Kumar +2
Distributed machine learning has become increasingly important due to the massive scale of large-scale generative models. Both model parameters and data are distributed across many…
How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee +9
Modern large language model (LLM) inference has progressively disaggregated to keep pace with growing model sizes and tight TTFT and TPOT service-level objectives: from chunked-pre…
MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces
Srinivas Sridharan, Theodor-Adrian Badea, Andy Balogh +26
The fast pace of artificial intelligence~(AI) innovation demands an agile methodology for observation, reproduction and optimization of distributed machine learning~(ML) workload b…
COSMIC: Enabling Full-Stack Co-Design and Optimization of Distributed Machine Learning Systems
Aditi Raju, Jared Ni, William Won +6
Large-scale machine learning models necessitate distributed systems, posing significant design challenges due to the large parameter space across distinct design stacks. Existing s…
Towards a Standardized Representation for Deep Learning Collective Algorithms
Jinsun Yoo, William Won, Meghan Cowan +4
The explosion of machine learning model size has led to its execution on distributed clusters at a very large scale. Many works have tried to optimize the process of producing coll…
FRED: Flexible REduction-Distribution Interconnect and Communication Implementation for Wafer-Scale Distributed Training of DNN Models
Saeed Rashidi, William Won, Sudarshan Srinivasan +2
Distributed Deep Neural Network (DNN) training is a technique to reduce the training overhead by distributing the training tasks into multiple accelerators, according to a parallel…
TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learning
William Won, Midhilesh Elavazhagan, Sudarshan Srinivasan +2
The surge of artificial intelligence, particularly large language models, has driven the rapid development of large-scale machine learning clusters. Executing distributed models on…
ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale
William Won, Taekyung Heo, Saeed Rashidi +3
As deep learning models and input data are scaling at an unprecedented rate, it is inevitable to move towards distributed training platforms to fit the model and increase training…
Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models
Saeed Rashidi, William Won, Sudarshan Srinivasan +2
Distributed training is a solution to reduce DNN training time by splitting the task across multiple NPUs (e.g., GPU/TPU). However, distributed training adds communication overhead…
LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Models
William Won, Saeed Rashidi, Sudarshan Srinivasan +1
As model sizes in machine learning continue to scale, distributed training is necessary to accommodate model weights within each device and to reduce training time. However, this c…
Extending Sparse Tensor Accelerators to Support Multiple Compression Formats
Eric Qin, Geonhwa Jeong, William Won +7
Sparsity, which occurs in both scientific applications and Deep Learning (DL) models, has been a key target of optimization within recent ASIC accelerators due to the potential mem…