12 citations · 21 across the 20 of their papers we have counts for
6 papers · 1 filter
Why Do We Need Warm-up? A Theoretical Perspective
Foivos Alimisis, Rustem Islamov, Aurelien Lucchi
Learning rate warm-up -- increasing the learning rate at the beginning of training -- has become a ubiquitous heuristic in modern deep learning, yet its theoretical foundations rem…
Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size
Rustem Islamov, Niccolo Ajroldi, Antonio Orvieto +1
Modern optimization algorithms that incorporate momentum and adaptive step-size offer improved performance in numerous challenging deep learning tasks. However, their effectiveness…
On the Interaction of Batch Noise, Adaptivity, and Compression, under -Smoothness: An SDE Approach
Enea Monzio Compagnoni, Rustem Islamov, Frank Norbert Proske +3
Distributed stochastic optimization intertwines (i) stochastic gradient noise, (ii) communication compression, and (iii) adaptive/normalized updates. While each factor has been stu…
Safe-EF: Error Feedback for Nonsmooth Constrained Optimization
Rustem Islamov, Yarden As, Ilyas Fatkhullin
Federated learning faces severe communication bottlenecks due to the high dimensionality of model updates. Communication compression with contractive compressors (e.g., Top-K) is o…
Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs
Enea Monzio Compagnoni, Rustem Islamov, Frank Norbert Proske +1
Distributed methods are essential for handling machine learning pipelines comprising large-scale models and datasets. However, their benefits often come at the cost of increased co…
Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy
Rustem Islamov, Samuel Horvath, Aurelien Lucchi +2
Strong Differential Privacy (DP) and Optimization guarantees are two desirable properties for a method in Federated Learning (FL). However, existing algorithms do not achieve both…