Adaptive Consensus Gradients Aggregation for Scaled Distributed Training

Yoni Choukroun, Shlomi Azoulay, Pavel Kisilev

Distributed machine learning has recently become a critical paradigm for training large models on vast datasets. We examine the stochastic optimization problem for deep learning wi…