4 papers
Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
Susav Shrestha, Brad Settlemyer, Nikoli Dryden +1
Accelerating large language model (LLM) inference is critical for real-world deployments requiring high throughput and low latency. Contextual sparsity, where each token dynamicall…
Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging
Shigang Li, Tal Ben-Nun, Giorgi Nadiradze +4
Deep learning at scale is dominated by communication time. Distributing samples across nodes usually yields the best performance, but poses scaling challenges due to global informa…
Lion Cub: Minimizing Communication Overhead in Distributed Lion
Satoki Ishikawa, Tal Ben-Nun, Brian Van Essen +2
Communication overhead is a key challenge in distributed deep learning, especially on slower Ethernet interconnects, and given current hardware trends, communication is likely to b…
Scaling Large-scale GNN Training to Thousands of Processors on CPU-based Supercomputers
Chen Zhuang, Lingqi Zhang, Du Wu +8
Graph Convolutional Networks (GCNs), particularly for large-scale graphs, are crucial across numerous domains. However, training distributed full-batch GCNs on large-scale graphs s…