1 citations · 2 across the 11 of their papers we have counts for
11 papers
Aligning Distributionally Robust Optimization with Practical Deep Learning Needs
Dmitrii Feoktistov, Igor Ignashin, Andrey Veprikov +4
While traditional Deep Learning (DL) optimization methods treat all training samples equally, Distributionally Robust Optimization (DRO) adaptively assigns importance weights to di…
Differentially Private Clipped-SGD: High-Probability Convergence with Arbitrary Clipping Level
Saleh Vatan Khah, Savelii Chezhegov, Shahrokh Farahmand +2
Gradient clipping is a fundamental tool in Deep Learning, improving the high-probability convergence of stochastic first-order methods like SGD, AdaGrad, and Adam under heavy-taile…
Incorporating Preconditioning into Accelerated Approaches: Theoretical Guarantees and Practical Improvement
Stepan Trifonov, Leonid Levin, Savelii Chezhegov +1
Machine learning and deep learning are widely researched fields that provide solutions to many modern problems. Due to the complexity of new problems related to the size of dataset…
High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and -Smoothness
Savelii Chezhegov, Taha El Bakkali El Kadi, Aleksandr Beznosikov +2
Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees over stochastic gradient descent (SGD) und…
Variance Reduction Methods Do Not Need to Compute Full Gradients: Improved Efficiency through Shuffling
Daniil Medyakov, Gleb Molodtsov, Savelii Chezhegov +2
Stochastic optimization algorithms are widely used for machine learning with large-scale data. However, their convergence often suffers from non-vanishing variance. Variance Reduct…
Local SGD for Near-Quadratic Problems: Improving Convergence under Unconstrained Noise Conditions
Andrey Sadchikov, Savelii Chezhegov, Aleksandr Beznosikov +1
Distributed optimization plays an important role in modern large-scale machine learning and data processing systems by optimizing the utilization of computational resources. One of…
Accelerated Stochastic Gradient Method with Applications to Consensus Problem in Markov-Varying Networks
Vladimir Solodkin, Savelii Chezhegov, Ruslan Nazikov +2
Stochastic optimization is a vital field in the realm of mathematical optimization, finding applications in diverse areas ranging from operations research to machine learning. In t…
Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
Savelii Chezhegov, Yaroslav Klyukin, Andrei Semenov +5
Methods with adaptive stepsizes, such as AdaGrad and Adam, are essential for training modern Deep Learning models, especially Large Language Models. Typically, the noise in the sto…
Decentralized Finite-Sum Optimization over Time-Varying Networks
Dmitry Metelev, Savelii Chezhegov, Alexander Rogozin +4
We consider decentralized time-varying stochastic optimization problems where each of the functions held by the nodes has a finite sum structure. Such problems can be efficiently s…
On Decentralized Nonsmooth Optimization
Savelii Chezhegov, Alexander Rogozin, Alexander Gasnikov
In decentralized optimization, several nodes connected by a network collaboratively minimize some objective function. For minimization of Lipschitz functions lower bounds are known…
A General Framework for Distributed Partitioned Optimization
Savelii Chezhegov, Anton Novitskii, Alexander Rogozin +3
Decentralized optimization is widely used in large scale and privacy preserving machine learning and various distributed control and sensing systems. It is assumed that every agent…