45 citations · 144 across the 35 of their papers we have counts for
6 papers · 1 filter
Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimization
Siddharth Singh, Zachary Sating, Abhinav Bhatele
Despite their better convergence properties compared to first-order optimizers, second-order optimizers for deep learning have been less popular due to their significant computatio…
HPC-Coder: Modeling Parallel Programs using Large Language Models
Daniel Nichols, Aniruddha Marathe, Harshitha Menon +2
Parallel programs in high performance computing (HPC) continue to grow in complexity and scale in the exascale era. The diversity in hardware and parallel programming models make d…
Pipit: Scripting the analysis of parallel execution traces
Abhinav Bhatele, Rakrish Dhakal, Alexander Movsesyan +2
Performance analysis is a critical step in the oft-repeated, iterative process of performance tuning of parallel programs. Per-process, per-thread traces (detailed logs of events w…
A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs
Siddharth Singh, Prajwal Singhania, Aditya K. Ranjan +2
Heavy communication, in particular, collective operations, can become a critical performance bottleneck in scaling the training of billion-parameter neural networks to large-scale…
A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan +3
Mixture-of-Experts (MoE) is a neural network architecture that adds sparsely activated expert blocks to a base model, increasing the number of parameters without impacting computat…
Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training
Siddharth Singh, Abhinav Bhatele
Parallel training of neural networks at scale is challenging due to significant overheads arising from communication. Recently, deep learning researchers have developed a variety o…