173 citations · 782 across the 61 of their papers we have counts for
24 papers · 1 filter
Robust Reasoning Benchmark
Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko +1
While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their problem-solving abilities depend on the context and textual formatting. We in…
LoRAFusion: Efficient LoRA Fine-Tuning for LLMs
Zhanda Zhu, Qidong Su, Yaoyao Ding +3
Low-Rank Adaptation (LoRA) has become the leading Parameter-Efficient Fine-Tuning (PEFT) method for Large Language Models (LLMs), as it significantly reduces GPU memory usage while…
DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
Yubo Gao, Renbo Tu, Gennady Pekhimenko +1
Differentially-Private SGD (DP-SGD) and its adaptive variant DP-Adam are powerful techniques to protect user privacy when using sensitive data to train neural networks. During trai…
Hexcute: A Compiler Framework for Automating Layout Synthesis in GPU Programs
Xiao Zhang, Yaoyao Ding, Bolin Sun +3
Efficient GPU programming is crucial for achieving high performance in deep learning (DL) applications. The performance of GPU programs depends on how data is parallelized across t…
Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation
Yaoyao Ding, Bohan Hou, Xiao Zhang +5
Serving Large Language Models (LLMs) is critical for AI-powered applications, yet it demands substantial computational resources, particularly in memory bandwidth and computational…
The Synergy of Speculative Decoding and Batching in Serving Large Language Models
Qidong Su, Christina Giannoula, Gennady Pekhimenko
Large Language Models (LLMs) like GPT are state-of-the-art text generation models that provide significant assistance in daily routines. However, LLM execution is inherently sequen…
Guaranteed Approximation Bounds for Mixed-Precision Neural Operators
Renbo Tu, Colin White, Jean Kossaifi +5
Neural operators, such as Fourier Neural Operators (FNO), form a principled approach for learning solution operators for PDEs and other mappings between function spaces. However, m…
Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction
Muralidhar Andoorveedu, Zhanda Zhu, Bojian Zheng +1
Training deep learning models can be computationally expensive. Prior works have shown that increasing the batch size can potentially lead to better overall throughput. However, th…
Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programs
Yaoyao Ding, Cody Hao Yu, Bojian Zheng +3
As deep learning models nowadays are widely adopted by both cloud services and edge devices, reducing the latency of deep learning model inferences becomes crucial to provide effic…
Optimizing Data Collection in Deep Reinforcement Learning
James Gleeson, Daniel Snider, Yvonne Yang +3
Reinforcement learning (RL) workloads take a notoriously long time to train due to the large number of samples collected at run-time from simulators. Unfortunately, cluster scale-u…
MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
Alexandros Karargyris, Renato Umeton, Micah J. Sheller +39
Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving prov…
Distributed Deep Learning in Open Collaborations
Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin +13
Modern deep learning applications require increasingly more compute to train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High…
Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models
Shang Wang, Peiming Yang, Yuxuan Zheng +2
Driven by the tremendous effort in researching novel deep learning (DL) algorithms, the training cost of developing new models increases staggeringly in recent years. We analyze GP…
RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads
James Gleeson, Srivatsan Krishnan, Moshe Gabel +3
Deep reinforcement learning (RL) has made groundbreaking advancements in robotics, data center management and other applications. Unfortunately, system-level bottlenecks in RL work…
Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk +1
Training deep neural networks on large datasets can often be accelerated by using multiple compute nodes. This approach, known as distributed training, can utilize hundreds of comp…
A Runtime-Based Computational Performance Predictor for Deep Neural Network Training
Geoffrey X. Yu, Yubo Gao, Pavel Golikov +1
Deep learning researchers and practitioners usually leverage GPUs to help train their deep neural networks (DNNs) faster. However, choosing which GPU to use is challenging both bec…
IOS: Inter-Operator Scheduler for CNN Acceleration
Yaoyao Ding, Ligeng Zhu, Zhihao Jia +2
To accelerate CNN inference, existing deep learning frameworks focus on optimizing intra-operator parallelization. However, a single operator can no longer fully utilize the availa…
Multi-node Bert-pretraining: Cost-efficient Approach
Jiahuang Lin, Xin Li, Gennady Pekhimenko
Recently, large scale Transformer-based language models such as BERT, GPT-2, and XLNet have brought about exciting leaps in state-of-the-art results for many Natural Language Proce…
MLPerf Inference Benchmark
Vijay Janapa Reddi, Christine Cheng, David Kanter +44
Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organization…
MLPerf Training Benchmark
Peter Mattson, Christine Cheng, Cody Coleman +34
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But M…
BPPSA: Scaling Back-propagation by Parallel Scan Algorithm
Shang Wang, Yifan Bai, Gennady Pekhimenko
In an era when the performance of a single compute device plateaus, software must be designed to scale on massively parallel systems for better runtime performance. However, in the…
MLSys: The New Frontier of Machine Learning Systems
Alexander Ratner, Dan Alistarh, Gustavo Alonso +66
Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains…
Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
Bojian Zheng, Abhishek Tiwari, Nandita Vijaykumar +1
The Long-Short-Term-Memory Recurrent Neural Networks (LSTM RNNs) are a popular class of machine learning models for analyzing sequential data. Their training on modern GPUs, howeve…
TBD: Benchmarking and Analyzing Deep Neural Network Training
Hongyu Zhu, Mohamed Akrout, Bojian Zheng +4
The recent popularity of deep neural networks (DNNs) has generated a lot of research interest in performing DNN-related computation efficiently. However, the primary focus is usual…