Accumulated Gradient Normalization

Joeri Hermans, Gerasimos Spanakis, Rico Möckel

This work addresses the instability in asynchronous data parallel optimization. It does so by introducing a novel distributed optimizer which is able to efficiently optimize a cent…