An Empirical Study on Compressed Decentralized Stochastic Gradient Algorithms with Overparameterized Models

Abstract

This paper considers decentralized optimization with application to machine learning on graphs. The growing size of neural network (NN) models has motivated prior works on decentralized stochastic gradient algorithms to incorporate communication compression. On the other hand, recent works have demonstrated the favorable convergence and generalization properties of overparameterized NNs. In this work, we present an empirical analysis on the performance of compressed decentralized stochastic gradient (DSG) algorithms with overparameterized NNs. Through simulations on an MPI network environment, we observe that the convergence rates of popular compressed DSG algorithms are robust to the size of NNs. Our findings suggest a gap between theories and practice of the compressed DSG algorithms in the existing literature.

arXiv comment7 pages, 6 figures, accepted to APSIPA 2021

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Communication Compression for Decentralized Training
• D${}^2$: Decentralized Training over Decentralized Data
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• An improved convergence analysis for decentralized online stochastic non-convex optimization
• Understanding Top-k Sparsification in Distributed Deep Learning