Collaborative Deep Learning Across Multiple Data Centers
arXiv:1810.06877
Abstract
Valuable training data is often owned by independent organizations and located in multiple data centers. Most deep learning approaches require to centralize the multi-datacenter data for performance purpose. In practice, however, it is often infeasible to transfer all data to a centralized data center due to not only bandwidth limitation but also the constraints of privacy regulations. Model averaging is a conventional choice for data parallelized training, but its ineffectiveness is claimed by previous studies as deep neural networks are often non-convex. In this paper, we argue that model averaging can be effective in the decentralized environment by using two strategies, namely, the cyclical learning rate and the increased number of epochs for local model training. With the two strategies, we show that model averaging can provide competitive performance in the decentralized mode compared to the data-centralized one. In a practical environment with multiple data centers, we conduct extensive experiments using state-of-the-art deep network architectures on different types of data. Results demonstrate the effectiveness and robustness of the proposed method.
arXiv commentSubmitted to AAAI 2019
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- Averaging Weights Leads to Wider Optima and Better Generalization
- Qualitatively characterizing neural network optimization problems
- Large scale distributed neural network training through online distillation
- Parallel training of DNNs with Natural Gradient and Parameter Averaging
- Multi-level Attention Model for Weakly Supervised Audio Classification
- Experiments on Parallel Training of Deep Neural Network using Model Averaging
- Towards Geo-Distributed Machine Learning