35 citations · 88 across the 18 of their papers we have counts for
19 papers
Optimizers for Diffusion Models: A Controlled Benchmark
Arman Bolatov, Egor Shulgin, David Li +6
Discrete diffusion models now match autoregressive language models on several benchmarks, while the question of how best to train them has received far less attention: the optimize…
Deriving Hyperparameter Scaling Laws via Modern Optimization Theory
Egor Shulgin, Dimitri von Rütte, Tianyue H. Zhang +3
Hyperparameter transfer has become an important component of modern large-scale training recipes. Existing methods, such as muP, primarily focus on transfer between model sizes, wi…
First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptions
Egor Shulgin, Grigory Malinovsky, Sarit Khirirat +1
Federated Learning (FL) enables collaborative training on decentralized data. Differential privacy (DP) is crucial for FL, but current private methods often rely on unrealistic ass…
Beyond the Ideal: Analyzing the Inexact Muon Update
Egor Shulgin, Sultan AlRashed, Francesco Orabona +1
The Muon optimizer has rapidly emerged as a powerful, geometry-aware alternative to AdamW, demonstrating strong performance in large-scale training of neural networks. However, a c…
Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)
Artem Riabinin, Egor Shulgin, Kaja Gruntkowska +1
Recent developments in deep learning optimization have brought about radically new algorithms based on the Linear Minimization Oracle (LMO) framework, such as and $\sf S…
Smoothed Normalization for Efficient Distributed Private Optimization
Egor Shulgin, Sarit Khirirat, Peter Richtárik
Federated learning enables training machine learning models while preserving the privacy of participants. Surprisingly, there is no differentially private distributed method for sm…
On the Convergence of DP-SGD with Adaptive Clipping
Egor Shulgin, Peter Richtárik
Stochastic Gradient Descent (SGD) with gradient clipping is a powerful technique for enabling differentially private optimization. Although prior works extensively investigated cli…
SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learning
Avetik Karagulyan, Egor Shulgin, Abdurakhmon Sadiev +1
Cross-device training is a crucial subfield of federated learning, where the number of clients can reach into the billions. Standard approaches and local methods are prone to issue…
MAST: Model-Agnostic Sparsified Training
Yury Demidovich, Grigory Malinovsky, Egor Shulgin +1
We introduce a novel optimization problem formulation that departs from the conventional way of minimizing machine learning model loss as a black-box function. Unlike traditional f…
Towards a Better Theoretical Understanding of Independent Subnetwork Training
Egor Shulgin, Peter Richtárik
Modern advancements in large-scale machine learning would be impossible without the paradigm of data-parallel distributed computing. Since distributed computing with large-scale mo…
Shifted Compression Framework: Generalizations and Improvements
Egor Shulgin, Peter Richtárik
Communication is one of the key bottlenecks in the distributed training of large-scale machine learning models, and lossy compression of exchanged information, such as stochastic g…
Certified Robustness in Federated Learning
Motasem Alfarra, Juan C. Pérez, Egor Shulgin +2
Federated learning has recently gained significant attention and popularity due to its effectiveness in training machine learning models on distributed data privately. However, as…
ADOM: Accelerated Decentralized Optimization Method for Time-Varying Networks
Dmitry Kovalev, Egor Shulgin, Peter Richtárik +2
We propose ADOM - an accelerated method for smooth and strongly convex decentralized optimization over time-varying networks. ADOM uses a dual oracle, i.e., we assume access to the…
Towards accelerated rates for distributed optimization over time-varying networks
Alexander Rogozin, Vladislav Lukoshkin, Alexander Gasnikov +2
We study the problem of decentralized optimization over time-varying networks with strongly convex smooth cost functions. In our approach, nodes run a multi-step gossip procedure a…
Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressor
Mher Safaryan, Egor Shulgin, Peter Richtárik
In order to mitigate the high communication cost in distributed and federated learning, various vector compression schemes, such as quantization, sparsification and dithering, have…
Adaptive Catalyst for Smooth Convex Optimization
Anastasiya Ivanova, Dmitry Pasechnyuk, Dmitry Grishchenko +3
In this paper, we present a generic framework that allows accelerating almost arbitrary non-accelerated deterministic and randomized algorithms for smooth convex optimization probl…
Lecture Notes on Stochastic Processes
Alexander Gasnikov, Eduard Gorbunov, Sergey Guz +3
This is lecture notes on the course "Stochastic Processes". In this format, the course was taught in the spring semesters 2017 and 2018 for third-year bachelor students of the Depa…
Revisiting Stochastic Extragradient
Konstantin Mishchenko, Dmitry Kovalev, Egor Shulgin +2
We fix a fundamental issue in the stochastic extragradient method by providing a new sampling strategy that is motivated by approximating implicit updates. Since the existing stoch…
SGD: General Analysis and Improved Rates
Robert Mansel Gower, Nicolas Loizou, Xun Qian +3
We propose a general yet simple theorem describing the convergence of SGD under the arbitrary sampling paradigm. Our theorem describes the convergence of an infinite array of varia…