63 citations · 119 across the 19 of their papers we have counts for
19 papers
OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction
Mohammad Mozaffari, Samuel Kushnir, Maryam Mehri Dehnavi +1
Post-training pruning cuts the inference cost of large language models, yet it faces a trade-off: simple heuristics that zero weights are fast but degrade accuracy, while principle…
Optimal Software Pipelining and Warp Specialization for Tensor Core GPUs
Rupanshu Soi, Rohan Yadav, Fredrik Kjolstad +4
GPU architectures have continued to grow in complexity, with recent incarnations introducing increasingly powerful fixed-function units for matrix multiplication and data movement…
PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
Younes Hourri, Mohammad Mozaffari, Maryam Mehri Dehnavi
Large language models (LLMs) deliver impressive performance but incur prohibitive memory and compute costs at deployment. Model pruning is an effective way to reduce these overhead…
SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Mohammad Mozaffari, Amir Yazdanbakhsh, Maryam Mehri Dehnavi
Conventional model compression techniques for LLMs address high memory consumption and slow inference challenges but typically require computationally expensive retraining to prese…
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs
Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang +1
We propose SLoPe, a Double-Pruned Sparse Plus Lazy Low-rank Adapter Pretraining method for LLMs that improves the accuracy of sparse LLMs while accelerating their pretraining and i…
MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates
Mohammad Mozaffari, Sikan Li, Zhao Zhang +1
This work proposes a Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 updates, called MKOR, that improves the training time and convergence properties of deep neural…
A Framework for Fine-Grained Synchronization of Dependent GPU Kernels
Abhinav Jangda, Saeed Maleki, Maryam Mehri Dehnavi +2
Machine Learning (ML) models execute several parallel computations including Generalized Matrix Multiplication, Convolution, Dropout, etc. These computations are commonly executed…
Vectorizing Sparse Matrix Codes with Dependency Driven Trace Analysis
Zachary Cetinic, Kazem Cheshmi, Maryam Mehri Dehnavi
Sparse computations frequently appear in scientific simulations and the performance of these simulations rely heavily on the optimization of the sparse codes. The compact data stru…
Composing Loop-carried Dependence with Other Loops
Kazem Cheshmi, Michelle Mills Strout, Maryam Mehri Dehnavi
Sparse fusion is a compile-time loop transformation and runtime scheduling implemented as a domain-specific code generator. Sparse fusion generates efficient parallel code for the…
L-DQN: An Asynchronous Limited-Memory Distributed Quasi-Newton Method
Bugra Can, Saeed Soori, Maryam Mehri Dehnavi +1
This work proposes a distributed algorithm for solving empirical risk minimization problems, called L-DQN, under the master/worker communication model. L-DQN is a distributed limit…
TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion
Saeed Soori, Bugra Can, Baourun Mu +2
This work proposes a time-efficient Natural Gradient Descent method, called TENGraD, with linear convergence guarantees. Computing the inverse of the neural network's Fisher inform…
Randomized Gossiping with Effective Resistance Weights: Performance Guarantees and Applications
Bugra Can, Saeed Soori, Necdet Serhat Aybat +2
The effective resistance between a pair of nodes in a weighted undirected graph is defined as the potential difference induced when a unit current is injected at one node and extra…
ASYNC: A Cloud Engine with Asynchrony and History for Distributed Machine Learning
Saeed Soori, Bugra Can, Mert Gurbuzbalaba +1
ASYNC is a framework that supports the implementation of asynchrony and history for optimization methods on distributed computing platforms. The popularity of asynchronous optimiza…
DAve-QN: A Distributed Averaged Quasi-Newton Method with Local Superlinear Convergence Rate
Saeed Soori, Konstantin Mischenko, Aryan Mokhtari +2
In this paper, we consider distributed algorithms for solving the empirical risk minimization problem under the master/worker communication model. We develop a distributed asynchro…
MatRox: Modular approach for improving data locality in Hierarchical (Mat)rix App(Rox)imation
Bangtian Liu, Kazem Cheshmi, Saeed Soori +2
Hierarchical matrix approximations have gained significant traction in the machine learning and scientific community as they exploit available low-rank structures in kernel methods…
Sparse Matrix Code Dependence Analysis Simplification at Compile Time
Mahdi Soltan Mohammadi, Kazem Cheshmi, Ganesh Gopalakrishnan +5
Analyzing array-based computations to determine data dependences is useful for many applications including automatic parallelization, race detection, computation and communication…
Avoiding Communication in Proximal Methods for Convex Optimization Problems
Saeed Soori, Aditya Devarakonda, James Demmel +2
The fast iterative soft thresholding algorithm (FISTA) is used to solve convex regularized optimization problems in machine learning. Distributed implementations of the algorithm h…
A Unified Optimization Approach for Sparse Tensor Operations on GPUs
Bangtian Liu, Chengyao Wen, Anand D. Sarwate +1
Sparse tensors appear in many large-scale applications with multidimensional and sparse data. While multidimensional sparse data often need to be processed on manycore processors,…
Sympiler: Transforming Sparse Matrix Codes by Decoupling Symbolic Analysis
Kazem Cheshmi, Shoaib Kamil, Michelle Mills Strout +1
Sympiler is a domain-specific code generator that optimizes sparse matrix computations by decoupling the symbolic analysis phase from the numerical manipulation stage in sparse cod…