12 papers
QK-Wanda: Coupling Queries and Keys for Unstructured Pruning
Ivan Ilin, Peter Richtárik
Wanda (Sun et al., 2024) prunes large language models by scoring weights independently within each linear projection, although queries and keys interact through dot products. We in…
Training Chemical Plausibility-Aware Large Language Models for Single-Step Retrosynthesis
Bogdan Zagribelnyy, Ivan Ilin, Nikita Bondarev +5
Single-step retrosynthesis is a central component of computer-aided synthesis planning, yet its intrinsically one-to-many nature is poorly captured by single-answer evaluation and…
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
Ivan Ilin, Philip Zmushko, Peter Richtárik
Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signa…
URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment
Bogdan Zagribelnyy, Ivan Ilin, Nikita Bondarev +7
Synthesis planning aiming to find pathways of reactions for a target molecule is one of the most important and challenging tasks in drug discovery. Recent progress has produced bot…
MMAI Gym for Science: Training Liquid Foundation Models for Drug Discovery
Maksim Kuznetsov, Zulfat Miftahutdinov, Rim Shayakhmetov +17
General-purpose large language models (LLMs) that rely on in-context learning do not reliably deliver the scientific understanding and performance required for drug discovery tasks…
When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs
Bogdan Zagribelnyy, Ivan Ilin, Maksim Kuznetsov +10
Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evaluation of retrosynthesis performance re…
Thanos: A Block-wise Pruning Algorithm for Efficient Large Language Model Compression
Ivan Ilin, Peter Richtarik
This paper presents Thanos, a novel weight-pruning algorithm designed to reduce the memory footprint and enhance the computational efficiency of large language models (LLMs) by rem…
Hessian of Perplexity for Large Language Models by PyTorch autograd (Open Source)
Ivan Ilin
Computing the full Hessian matrix -- the matrix of second-order derivatives for an entire Large Language Model (LLM) is infeasible due to its sheer size. In this technical report,…
Pushing the Limits of Large Language Model Quantization via the Linearity Theorem
Vladimir Malinovskii, Andrei Panferov, Ivan Ilin +3
Quantizing large language models has become a standard way to reduce their memory and computational costs. Typically, existing methods focus on breaking down the problem into indiv…
PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
Vladimir Malinovskii, Denis Mazur, Ivan Ilin +5
There has been significant interest in "extreme" compression of large language models (LLMs), i.e., to 1-2 bits per parameter, which allows such models to be executed efficiently o…
Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneity
Alexander Tyurin, Marta Pozzi, Ivan Ilin +1
We consider nonconvex stochastic optimization problems in the asynchronous centralized distributed setup where the communication times from workers to a server can not be ignored,…
Kimad: Adaptive Gradient Compression with Bandwidth Awareness
Jihao Xin, Ivan Ilin, Shunkang Zhang +2
In distributed training, communication often emerges as a bottleneck. In response, we introduce Kimad, a solution that offers adaptive gradient compression. By consistently monitor…