492 citations · 1.1k across the 52 of their papers we have counts for
7 papers · 1 filter
On Optimizing the Communication of Model Parallelism
Yonghao Zhuang, Hexu Zhao, Lianmin Zheng +6
We study a novel and important communication pattern in large-scale model-parallel deep learning (DL), which we call cross-mesh resharding. This pattern emerges when the two paradi…
Skyplane: Optimizing Transfer Cost and Throughput Using Cloud-Aware Overlays
Paras Jain, Sam Kumar, Sarah Wooders +3
Cloud applications are increasingly distributing data across multiple regions and cloud providers. Unfortunately, wide-area bulk data transfers are often slow, bottlenecking applic…
Context-Aware Streaming Perception in Dynamic Environments
Gur-Eyal Sela, Ionel Gog, Justin Wong +9
Efficient vision works maximize accuracy under a latency budget. These works evaluate accuracy offline, one image at a time. However, real-time vision applications like autonomous…
POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging
Shishir G. Patil, Paras Jain, Prabal Dutta +2
Fine-tuning models on edge devices like mobile phones would enable privacy-preserving personalization over sensitive data. However, edge training has historically been limited to r…
NumS: Scalable Array Programming for the Cloud
Melih Elibol, Vinamra Benara, Samyu Yagati +4
Scientists increasingly rely on Python tools to perform scalable distributed memory array operations using rich, NumPy-like expressions. However, many of these tools rely on dynami…
Representing Long-Range Context for Graph Neural Networks with Global Attention
Zhanghao Wu, Paras Jain, Matthew A. Wright +3
Graph neural networks are powerful architectures for structured datasets. However, current methods struggle to represent long-range dependencies. Scaling the depth or width of GNNs…