Beyond Gradient Averaging in Parallel Optimization: Improved Robustness through Gradient Agreement Filtering

Francois Chaubard, Duncan Eddy, Mykel J. Kochenderfer

We introduce Gradient Agreement Filtering (GAF) to improve on gradient averaging in distributed deep learning optimization. Traditional distributed data-parallel stochastic gradien…