21 citations · 136 across the 22 of their papers we have counts for
4 papers · 1 filter
Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits
Daniel Morales-Brotons, Thijs Vogels, Hadrien Hendrikx
Weight averaging of Stochastic Gradient Descent (SGD) iterates is a popular method for training deep learning models. While it is often used as part of complex training pipelines t…
Unified Breakdown Analysis for Byzantine Robust Gossip
Renaud Gaucher, Aymeric Dieuleveut, Hadrien Hendrikx
In decentralized machine learning, different devices communicate in a peer-to-peer manner to collaboratively learn from each other's data. Such approaches are vulnerable to misbeha…
Byzantine-Robust Gossip: Insights from a Dual Approach
Renaud Gaucher, Aymeric Dieuleveut, Hadrien Hendrikx
Distributed learning has many computational benefits but is vulnerable to attacks from a subset of devices transmitting incorrect information. This paper investigates Byzantine-res…
Investigating Variance Definitions for Mirror Descent with Relative Smoothness
Hadrien Hendrikx
Mirror Descent is a popular algorithm, that extends Gradients Descent (GD) beyond the Euclidean geometry. One of its benefits is to enable strong convergence guarantees through smo…