7 citations · 9 across the 4 of their papers we have counts for
4 papers
CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems
Tingkai Liu, Muralidhar Andoorveedu, Sanjoy Das +2
The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated shared-memory structures. However, current software still mostly treats these cohe…
Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization
Zhanda Zhu, Christina Giannoula, Muralidhar Andoorveedu +4
Various parallelism, such as data, tensor, and pipeline parallelism, along with memory optimizations like activation checkpointing, redundancy elimination, and offloading, have bee…
Seesaw: High-throughput LLM Inference via Model Re-sharding
Qidong Su, Wei Zhao, Xin Li +6
To improve the efficiency of distributed large language model (LLM) inference, various parallelization strategies, such as tensor and pipeline parallelism, have been proposed. Howe…
Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction
Muralidhar Andoorveedu, Zhanda Zhu, Bojian Zheng +1
Training deep learning models can be computationally expensive. Prior works have shown that increasing the batch size can potentially lead to better overall throughput. However, th…