Efficient and Private Federated Learning with Partially Trainable Networks
arXiv:2110.03450
Abstract
Federated learning is used for decentralized training of machine learning models on a large number (millions) of edge mobile devices. It is challenging because mobile devices often have limited communication bandwidth and local computation resources. Therefore, improving the efficiency of federated learning is critical for scalability and usability. In this paper, we propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process, to reduce the communication cost with little implications on model performance. Through extensive experiments, we empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs, with up to reduction in communication cost, at a small accuracy cost. Our approach also enables faster training, with a smaller memory footprint, and better utility for strong differential privacy guarantees. The proposed FedPT method can be particularly interesting for pushing the limitations of over-parameterization in on-device learning.
V2: minor wording and format improvement
References in corpus (26)
- Distilling the Knowledge in a Neural Network
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- Do Deep Nets Really Need to be Deep?
- Towards Federated Learning at Scale: System Design
- Learning Differentially Private Recurrent Language Models
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- Generating Long Sequences with Sparse Transformers
- EMNIST: an extension of MNIST to handwritten letters
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- LEAF: A Benchmark for Federated Settings
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
- A Field Guide to Federated Optimization
- Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
- A Practical Survey on Faster and Lighter Transformers
- Adaptive Federated Optimization
- Rethinking Attention with Performers
- Are All Layers Created Equal?
- Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
- No Training Required: Exploring Random Encoders for Sentence Classification
- FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
- Federated Reconstruction: Partially Local Federated Learning
- Training Production Language Models without Memorizing User Data
- Echo State Neural Machine Translation
- Pufferfish: Communication-efficient Models At No Extra Cost