1 citations · 1 across the 8 of their papers we have counts for
7 papers · 1 filter
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
Ivan Ilin, Philip Zmushko, Peter Richtárik
Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signa…
One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining
Philip Zmushko, Egor Petrov, Nursultan Abdullaev +2
Modern large-scale LLM pretraining benefits from utilizing Pipeline Parallelism; however, synchronous implementations leave GPUs idle during pipeline bubbles, wasting computational…
DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers
Ionut-Vlad Modoranu, Philip Zmushko, Erik Schultheis +2
Shampoo is one of the leading approximate second-order optimizers: a variant of it has won the MLCommons AlgoPerf competition, and it has been shown to produce models with lower ac…
Sign-SGD via Parameter-Free Optimization
Daniil Medyakov, Sergey Stanko, Gleb Molodtsov +4
Large language models have achieved major advances across domains, yet training them remains extremely resource-intensive. We revisit Sign-SGD, which serves both as a memory-effici…
Label Privacy in Split Learning for Large Models with Parameter-Efficient Training
Philip Zmushko, Marat Mansurov, Ruslan Svirschevski +3
As deep learning models become larger and more expensive, many practitioners turn to fine-tuning APIs. These web services allow fine-tuning a model between two parties: the client…
Just a Simple Transformation is Enough for Data Protection in Vertical Federated Learning
Andrei Semenov, Philip Zmushko, Alexander Pichugin +1
Vertical Federated Learning (VFL) aims to enable collaborative training of deep learning models while maintaining privacy protection. However, the VFL procedure still has component…
FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training
Philip Zmushko, Aleksandr Beznosikov, Martin Takáč +1
With the increase in the number of parameters in large language models, the process of pre-training and fine-tuning increasingly demands larger volumes of GPU memory. A significant…