39 citations · 42 across the 7 of their papers we have counts for
7 papers
Plexus: Taming Billion-edge Graphs with 3D Parallel Full-graph GNN Training
Aditya K. Ranjan, Siddharth Singh, Cunyang Wei +1
Graph neural networks (GNNs) leverage the connectivity and structure of real-world graphs to learn intricate properties and relationships between nodes. Many real-world graphs exce…
HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages
Aman Chaturvedi, Daniel Nichols, Siddharth Singh +1
Large Language Model (LLM) based coding tools have been tremendously successful as software development assistants, yet they are often designed for general purpose programming task…
Loki: Low-rank Keys for Efficient Sparse Attention
Prajwal Singhania, Siddharth Singh, Shwai He +2
Inference on large language models (LLMs) can be expensive in terms of the compute and memory costs involved, especially when long sequence lengths are used. In particular, the sel…
Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimization
Siddharth Singh, Zachary Sating, Abhinav Bhatele
Despite their better convergence properties compared to first-order optimizers, second-order optimizers for deep learning have been less popular due to their significant computatio…
A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs
Siddharth Singh, Prajwal Singhania, Aditya K. Ranjan +2
Heavy communication, in particular, collective operations, can become a critical performance bottleneck in scaling the training of billion-parameter neural networks to large-scale…
A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan +3
Mixture-of-Experts (MoE) is a neural network architecture that adds sparsely activated expert blocks to a base model, increasing the number of parameters without impacting computat…
Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training
Siddharth Singh, Abhinav Bhatele
Parallel training of neural networks at scale is challenging due to significant overheads arising from communication. Recently, deep learning researchers have developed a variety o…