1 citations · 1 across the 6 of their papers we have counts for
7 papers
ROTE: Benchmarking Neural Memorization on Complexity-Controlled Symbolic Sequences
Xinye Chen, Stefan Güttel, Mohammad Mozaffari
We introduce ROTE (RollOut Testing of Exact memorization), a benchmarking protocol for evaluating symbolic memorization of neural architectures. We study memorization and the exten…
Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression
Mohammad Mozaffari
Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models (LLMs). Traditional compression techniques (sparsity, quantization, low-ra…
LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models
Mohammad Mozaffari, Younes Hourri, Mohammad Rastegari +1
Unstructured sparsity is now natively accelerated by recent GPU kernels and dataflow hardware, shifting the bottleneck from inference execution to the pruning algorithm. State-of-t…
OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction
Mohammad Mozaffari, Samuel Kushnir, Maryam Mehri Dehnavi +1
Post-training pruning cuts the inference cost of large language models, yet it faces a trade-off: simple heuristics that zero weights are fast but degrade accuracy, while principle…
PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
Younes Hourri, Mohammad Mozaffari, Maryam Mehri Dehnavi
Large language models (LLMs) deliver impressive performance but incur prohibitive memory and compute costs at deployment. Model pruning is an effective way to reduce these overhead…
SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Mohammad Mozaffari, Amir Yazdanbakhsh, Maryam Mehri Dehnavi
Conventional model compression techniques for LLMs address high memory consumption and slow inference challenges but typically require computationally expensive retraining to prese…
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs
Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang +1
We propose SLoPe, a Double-Pruned Sparse Plus Lazy Low-rank Adapter Pretraining method for LLMs that improves the accuracy of sparse LLMs while accelerating their pretraining and i…