71 citations · 151 across the 51 of their papers we have counts for
9 papers · 1 filter
Tame Riemannian Stochastic Approximation
Johannes Aspman, Vyacheslav Kungurtsev, Reza Roohi Seraji
We study the properties of stochastic approximation applied to a tame nondifferentiable function subject to constraints defined by a Riemannian manifold. The objective landscape of…
When Do Curricula Work in Federated Learning?
Saeed Vahidian, Sreevatsank Kadaveru, Woonjoon Baek +5
An oft-cited open problem of federated learning is the existence of data heterogeneity at the clients. One pathway to understanding the drastic accuracy drop in federated learning…
Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergence
Diyuan Wu, Vyacheslav Kungurtsev, Marco Mondelli
The stochastic heavy ball method (SHB), also known as stochastic gradient descent (SGD) with Polyak's momentum, is widely used in training neural networks. However, despite the rem…
Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces
Saeed Vahidian, Mahdi Morafah, Weijia Wang +4
Clustered federated learning (FL) has been shown to produce promising results by grouping clients into clusters. This is especially effective in scenarios where separate groups of…
Scaling the Wild: Decentralizing Hogwild!-style Shared-memory SGD
Bapi Chatterjee, Vyacheslav Kungurtsev, Dan Alistarh
Powered by the simplicity of lock-free asynchrony, Hogwilld! is a go-to approach to parallelize SGD over a shared-memory setting. Despite its popularity and concomitant extensions,…
Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
Alexander Shevchenko, Vyacheslav Kungurtsev, Marco Mondelli
Understanding the properties of neural networks trained via stochastic gradient descent (SGD) is at the heart of the theory of deep learning. In this work, we take a mean-field vie…
Decentralized Bayesian Learning with Metropolis-Adjusted Hamiltonian Monte Carlo
Vyacheslav Kungurtsev, Adam Cobb, Tara Javidi +1
Federated learning performed by a decentralized networks of agents is becoming increasingly important with the prevalence of embedded software on autonomous devices. Bayesian appro…
Stochastic Gradient Langevin with Delayed Gradients
Vyacheslav Kungurtsev, Bapi Chatterjee, Dan Alistarh
Stochastic Gradient Langevin Dynamics (SGLD) ensures strong guarantees with regards to convergence in measure for sampling log-concave posterior distributions by adding noise to st…
Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent
Giorgi Nadiradze, Ilia Markov, Bapi Chatterjee +2
Machine learning has made tremendous progress in recent years, with models matching or even surpassing humans on a series of specialized tasks. One key element behind the progress…