1 citations · 1 across the 10 of their papers we have counts for
10 papers
Enhancing LLM Training via Spectral Clipping
Xiaowen Jiang, Andrei Semenov, Sebastian U. Stich
While spectral-based optimizers like Muon operate directly on the spectrum of updates, standard adaptive methods such as AdamW do not account for the spectral structure of weights…
Adaptive Regularized Newton Method with Inexact Hessian
Aleksandr Shestakov, Nail Bashirov, Andrei Semenov +4
Newton's method is the most widespread high-order method, demanding the gradient and the Hessian of the objective function. However, one of the main disadvantages of Newtons method…
Benchmarking Optimizers for Large Language Model Pretraining
Andrei Semenov, Matteo Pagliardini, Martin Jaggi
The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Clai…
Gradient-Normalized Smoothness for Optimization with Approximate Hessians
Andrei Semenov, Martin Jaggi, Nikita Doikov
In this work, we develop new optimization algorithms that use approximate second-order information combined with the gradient regularization technique to achieve fast global conver…
Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under -Smoothness
Nikita Kornilov, Philip Zmushko, Andrei Semenov +3
In recent years, non-convex optimization problems are more often described by generalized -smoothness assumption rather than standard one. Meanwhile, severely corrupted…
Just a Simple Transformation is Enough for Data Protection in Vertical Federated Learning
Andrei Semenov, Philip Zmushko, Alexander Pichugin +1
Vertical Federated Learning (VFL) aims to enable collaborative training of deep learning models while maintaining privacy protection. However, the VFL procedure still has component…
Mixed Newton Method for Optimization in Complex Spaces
Nikita Yudin, Roland Hildebrand, Sergey Bakhurin +5
In this paper, we modify and apply the recently introduced Mixed Newton Method, which is originally designed for minimizing real-valued functions of complex variables, to the minim…
Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
Savelii Chezhegov, Yaroslav Klyukin, Andrei Semenov +5
Methods with adaptive stepsizes, such as AdaGrad and Adam, are essential for training modern Deep Learning models, especially Large Language Models. Typically, the noise in the sto…
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
Andrei Semenov, Vladimir Ivanov, Aleksandr Beznosikov +1
We propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs). While SOTA approaches to Image Classification task work as a black b…
Bregman Proximal Method for Efficient Communications under Similarity
Aleksandr Beznosikov, Darina Dvinskikh, Dmitry Bylinkin +2
We propose a novel stochastic distributed method for both monotone and strongly monotone variational inequalities with Lipschitz operator and proper convex regularizers arising in…