4 citations · 9 across the 7 of their papers we have counts for
9 papers
QK-Wanda: Coupling Queries and Keys for Unstructured Pruning
Ivan Ilin, Peter Richtárik
Wanda (Sun et al., 2024) prunes large language models by scoring weights independently within each linear projection, although queries and keys interact through dot products. We in…
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
Ivan Ilin, Philip Zmushko, Peter Richtárik
Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signa…
Thanos: A Block-wise Pruning Algorithm for Efficient Large Language Model Compression
Ivan Ilin, Peter Richtarik
This paper presents Thanos, a novel weight-pruning algorithm designed to reduce the memory footprint and enhance the computational efficiency of large language models (LLMs) by rem…
PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
Vladimir Malinovskii, Denis Mazur, Ivan Ilin +5
There has been significant interest in "extreme" compression of large language models (LLMs), i.e., to 1-2 bits per parameter, which allows such models to be executed efficiently o…
MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
Ionut-Vlad Modoranu, Mher Safaryan, Grigory Malinovsky +4
We propose a new variant of the Adam optimizer called MicroAdam that specifically minimizes memory overheads, while maintaining theoretical convergence guarantees. We achieve this…
Kimad: Adaptive Gradient Compression with Bandwidth Awareness
Jihao Xin, Ivan Ilin, Shunkang Zhang +2
In distributed training, communication often emerges as a bottleneck. In response, we introduce Kimad, a solution that offers adaptive gradient compression. By consistently monitor…
Federated Learning is Better with Non-Homomorphic Encryption
Konstantin Burlachenko, Abdulmajeed Alrowithi, Fahad Ali Albalawi +1
Traditional AI methodologies necessitate centralized data collection, which becomes impractical when facing problems with network communication, data privacy, or storage capacity.…
Consensus-Based Optimization with Truncated Noise
Massimo Fornasier, Peter Richtárik, Konstantin Riedl +1
Consensus-based optimization (CBO) is a versatile multi-particle metaheuristic optimization method suitable for performing nonconvex and nonsmooth global optimizations in high dime…
Faster Rates for Compressed Federated Learning with Client-Variance Reduction
Haoyu Zhao, Konstantin Burlachenko, Zhize Li +1
Due to the communication bottleneck in distributed and federated learning applications, algorithms using communication compression have attracted significant attention and are wide…