4 papers
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov +2
Sign-based and LMO-inspired optimizers have recently attracted substantial attention in deep learning due to their strong performance and low memory footprint. However, their fixed…
Benchmarking Optimizers for MLPs in Tabular Deep Learning
Yury Gorishniy, Ivan Rubachev, Dmitrii Feoktistov +1
MLP is a heavily used backbone in modern deep learning (DL) architectures for supervised learning on tabular data, and AdamW is the go-to optimizer used to train tabular DL models.…
Aligning Distributionally Robust Optimization with Practical Deep Learning Needs
Dmitrii Feoktistov, Igor Ignashin, Andrey Veprikov +4
While traditional Deep Learning (DL) optimization methods treat all training samples equally, Distributionally Robust Optimization (DRO) adaptively assigns importance weights to di…
Lower Bounds and Optimal Algorithms for Non-Smooth Convex Decentralized Optimization over Time-Varying Networks
Dmitry Kovalev, Ekaterina Borodich, Alexander Gasnikov +1
We consider the task of minimizing the sum of convex functions stored in a decentralized manner across the nodes of a communication network. This problem is relatively well-studied…