48 citations · 72 across the 17 of their papers we have counts for
14 papers · 1 filter
From Skew to Symmetry: Node-Interconnect Multi-Path Balancing with Execution-time Planning for Modern GPU Clusters
Jinghan Yao, Kaushik Kandadi, Bharath Ramesh +2
Modern GPU-based high-performance computing clusters offer unprecedented communication bandwidth through heterogeneous intra-node interconnects and inter-node networks. However, de…
Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning
Lang Xu, Quentin Anthony, Jacob Hatef +4
Scaling up Large Language Model(LLM) training involves fitting a tremendous amount of training parameters across a limited number of workers. However, methods like ZeRO-3 that dras…
Accelerating Large Language Model Training with Hybrid GPU-based Compression
Lang Xu, Quentin Anthony, Qinghua Zhou +5
Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) are the three strategies widely adopted to enable fast and efficient Large Language Model (LLM) traini…
Demystifying the Communication Characteristics for Distributed Transformer Models
Quentin Anthony, Benjamin Michalowicz, Jacob Hatef +5
Deep learning (DL) models based on the transformer architecture have revolutionized many DL applications such as large language models (LLMs), vision transformers, audio generation…
Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer
Jinghan Yao, Sam Ade Jacobs, Masahiro Tanaka +3
Large Language Models (LLMs) with long context capabilities are integral to complex tasks in natural language processing and computational biology, such as text generation and prot…
The Case for Co-Designing Model Architectures with Hardware
Quentin Anthony, Jacob Hatef, Deepak Narayanan +6
While GPUs are responsible for training the vast majority of state-of-the-art deep learning models, the implications of their architecture are often overlooked when designing new d…
Flover: A Temporal Fusion Framework for Efficient Autoregressive Model Parallel Inference
Jinghan Yao, Nawras Alnaasan, Tian Chen +4
Autoregressive models, despite their commendable performance in a myriad of generative tasks, face challenges stemming from their inherently sequential structure. Inference on thes…
MCR-DL: Mix-and-Match Communication Runtime for Deep Learning
Quentin Anthony, Ammar Ahmad Awan, Jeff Rasley +5
In recent years, the training requirements of many state-of-the-art Deep Learning (DL) models have scaled beyond the compute and memory capabilities of a single processor, and nece…
OMB-Py: Python Micro-Benchmarks for Evaluating Performance of MPI Libraries on HPC Systems
Nawras Alnaasan, Arpan Jain, Aamir Shafi +2
Python has become a dominant programming language for emerging areas like Machine Learning (ML), Deep Learning (DL), and Data Science (DS). An attractive feature of Python is that…
Efficient MPI-based Communication for GPU-Accelerated Dask Applications
Aamir Shafi, Jahanzeb Maqbool Hashmi, Hari Subramoni +1
Dask is a popular parallel and distributed computing framework, which rivals Apache Spark to enable task-based scalable processing of big data. The Dask Distributed library forms t…
HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow
Ammar Ahmad Awan, Arpan Jain, Quentin Anthony +2
To reduce training time of large-scale DNNs, scientists have started to explore parallelization strategies like data-parallelism, model-parallelism, and hybrid-parallelism. While d…
Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation
Ammar Ahmad Awan, Jeroen Bedorf, Ching-Hsiang Chu +2
TensorFlow has been the most widely adopted Machine/Deep Learning framework. However, little exists in the literature that provides a thorough understanding of the capabilities whi…
Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?
Ammar Ahmad Awan, Ching-Hsiang Chu, Hari Subramoni +1
Dense Multi-GPU systems have recently gained a lot of attention in the HPC arena. Traditionally, MPI runtimes have been primarily designed for clusters with a large number of nodes…
System-level Scalable Checkpoint-Restart for Petascale Computing
Jiajun Cao, Kapil Arya, Rohan Garg +5
Fault tolerance for the upcoming exascale generation has long been an area of active research. One of the components of a fault tolerance strategy is checkpointing. Petascale-level…