AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning

Abstract

In multi-party collaborative learning, the parameter server sends a global model to each data holder for local training and then aggregates committed models globally to achieve privacy protection. However, both the dragger issue of synchronous collaborative learning and the staleness issue of asynchronous collaborative learning make collaborative learning inefficient in real-world heterogeneous environments. We propose a novel and efficient collaborative learning framework named AdaptCL, which generates an adaptive sub-model dynamically from the global base model for each data holder, without any prior information about worker capability. All workers (data holders) achieve approximately identical update time as the fastest worker by equipping them with capability-adapted pruned models. Thus the training process can be dramatically accelerated. Besides, we tailor the efficient pruned rate learning algorithm and pruning approach for AdaptCL. Meanwhile, AdaptCL provides a mechanism for handling the trade-off between accuracy and time overhead and can be combined with other techniques to accelerate training further. Empirical results show that AdaptCL introduces little computing and communication overhead. AdaptCL achieves time savings of more than 41\% on average and improves accuracy in a low heterogeneous environment. In a highly heterogeneous environment, AdaptCL achieves a training speedup of 6.2x with a slight loss of accuracy.

References in corpus (22)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Pruning Filters for Efficient ConvNets
• Asynchronous Federated Optimization
• Understanding the Role of Individual Units in a Deep Neural Network
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Asynchronous Stochastic Gradient Descent with Delay Compensation
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• High-Performance Distributed ML at Scale through Parameter Server Consistency Models
• Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
• Toward Understanding the Impact of Staleness in Distributed Machine Learning
• Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning

Cited by in corpus (1)

• Boosting Resource-Constrained Federated Learning Systems with Guessed Updates