45 citations · 144 across the 35 of their papers we have counts for
40 papers
SpCCL: A Sparsity-Aware Collective Communication Library for GPU Platforms
Lannie Dalton Hough, Emir Gencer, Hoffmann Muki +1
Collective communication is essential to high performance computing and machine learning workloads, yet libraries such as NCCL do not exploit sparsity in message payloads. Sending…
KEET: Explaining Performance of GPU Kernels Using LLM Agents
Joshua H. Davis, Klaudiusz Rydzy, Srinivasan Ramesh +5
Performance profiles of GPU kernels generated by tools such as Nsight Compute are rich in detail but are often challenging to interpret. To achieve the best performance possible on…
Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Training
Cunyang Wei, Siddharth Singh, Aishwarya Sarkar +7
Graph neural networks (GNNs) are widely used for learning on graph datasets derived from various real-world scenarios. Learning from extremely large graphs requires distributed tra…
Speculating Experts Accelerates Inference for Mixture-of-Experts
Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token…
Optimizing Agentic Language Model Inference via Speculative Tool Calls
Daniel Nichols, Prajwal Singhania, Charles Jekel +2
Language models (LMs) are becoming increasingly dependent on external tools. LM-based agentic frameworks frequently interact with their environment via such tools to search files,…
Understanding and Improving Communication Performance in Multi-node LLM Inference
Prajwal Singhania, Siddharth Singh, Lannie Dalton Hough +4
As large language models (LLMs) continue to grow in size, distributed inference has become increasingly important. Model-parallel strategies must now efficiently scale not only acr…
Integrating Performance Tools in Model Reasoning for GPU Kernel Optimization
Daniel Nichols, Konstantinos Parasyris, Charles Jekel +2
Language models are now prevalent in software engineering with many developers using them to automate tasks and accelerate their development. While language models have been tremen…
Modeling Code: Is Text All You Need?
Daniel Nichols, Konstantinos Parasyris, Harshitha Menon +4
Code LLMs have become extremely popular recently for modeling source code across a variety of tasks, such as generation, translation, and summarization. However, transformer-based…
Power Law Guided Dynamic Sifting for Efficient Attention
Nirav Koley, Prajwal Singhania, Abhinav Bhatele
Efficient inference on GPUs using large language models remains challenging due to memory bandwidth limitations, particularly during data transfers between High Bandwidth Memory (H…
ParEval-Repo: A Benchmark Suite for Evaluating LLMs with Repository-level HPC Translation Tasks
Joshua H. Davis, Daniel Nichols, Ishan Khillan +1
GPGPU architectures have become significantly more diverse in recent years, which has led to an emergence of a variety of specialized programming models and software stacks to supp…
Simulating nationwide coupled disease and fear spread in an agent-based model
Joy Kitson, Prescott C. Alexander, Joseph Tuccillo +5
Human cognitive responses, behavioral responses, and disease dynamics co-evolve over the course of any disease outbreak, and can result in complex feedbacks. We present a dynamic a…
Zero-Shot Vision Encoder Grafting via LLM Surrogates
Kaiyu Yue, Vasu Singla, Menglin Jia +6
Vision language models (VLMs) typically pair a modestly sized vision encoder with a large language model (LLM), e.g., Llama-70B, making the decoder the primary computational burden…
Plexus: Taming Billion-edge Graphs with 3D Parallel Full-graph GNN Training
Aditya K. Ranjan, Siddharth Singh, Cunyang Wei +1
Graph neural networks (GNNs) leverage the connectivity and structure of real-world graphs to learn intricate properties and relationships between nodes. Many real-world graphs exce…
The Big Send-off: Scalable and Performant Collectives for Deep Learning
Siddharth Singh, Keshav Pradeep, Mahua Singh +2
Collective communication is becoming increasingly important in data center and supercomputer workloads with an increase in distributed AI related jobs. However, existing libraries…
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Jonas Geiping, Sean McLeish, Neel Jain +6
We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent bloc…
Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers
Siddharth Singh, Prajwal Singhania, Aditya Ranjan +9
Training and fine-tuning large language models (LLMs) with hundreds of billions to trillions of parameters requires tens of thousands of GPUs, and a highly scalable software stack.…
Characterizing Production GPU Workloads using System-wide Telemetry Data
Onur Cankur, Brian Austin, Dhruva Kulkarni +1
GPGPU-accelerated clusters and supercomputers are central to modern high-performance computing (HPC). Over the past decade, these systems continue to expand, and GPUs now expose a…
Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Sean McLeish, John Kirchenbauer, David Yu Miller +5
Scaling laws are typically fit using a family of models with a narrow range of frozen hyperparameter choices. In this work we study scaling laws using multiple architectural shapes…
HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages
Aman Chaturvedi, Daniel Nichols, Siddharth Singh +1
Large Language Model (LLM) based coding tools have been tremendously successful as software development assistants, yet they are often designed for general purpose programming task…
From Pixels to Prose: A Large Dataset of Dense Image Captions
Vasu Singla, Kaiyu Yue, Sukriti Paul +7
Training large vision-language models requires extensive, high-quality image-text pairs. Existing web-scraped datasets, however, are noisy and lack detailed image descriptions. To…
Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs
Abhimanyu Hans, Yuxin Wen, Neel Jain +8
Large language models can memorize and repeat their training data, causing privacy and copyright risks. To mitigate memorization, we introduce a subtle modification to the next-tok…
Loki: Low-rank Keys for Efficient Sparse Attention
Prajwal Singhania, Siddharth Singh, Shwai He +2
Inference on large language models (LLMs) can be expensive in terms of the compute and memory costs involved, especially when long sequence lengths are used. In particular, the sel…
Transformers Can Do Arithmetic with the Right Embeddings
Sean McLeish, Arpit Bansal, Alex Stein +8
The poor performance of transformers on arithmetic tasks seems to stem in large part from their inability to keep track of the exact position of each digit inside of a large span o…
Performance-Aligned LLMs for Generating Fast Code
Daniel Nichols, Pranav Polasam, Harshitha Menon +3
Optimizing scientific software is a difficult task because codebases are often large and complex, and performance can depend upon several factors including the algorithm, its imple…