Efficient and Private Federated Learning with Partially Trainable Networks

Abstract

Federated learning is used for decentralized training of machine learning models on a large number (millions) of edge mobile devices. It is challenging because mobile devices often have limited communication bandwidth and local computation resources. Therefore, improving the efficiency of federated learning is critical for scalability and usability. In this paper, we propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process, to reduce the communication cost with little implications on model performance. Through extensive experiments, we empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs, with up to $46\times$ reduction in communication cost, at a small accuracy cost. Our approach also enables faster training, with a smaller memory footprint, and better utility for strong differential privacy guarantees. The proposed FedPT method can be particularly interesting for pushing the limitations of over-parameterization in on-device learning.

V2: minor wording and format improvement

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Towards Federated Learning at Scale: System Design
• Generating Long Sequences with Sparse Transformers
• EMNIST: an extension of MNIST to handwritten letters
• A Field Guide to Federated Optimization
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• No Training Required: Exploring Random Encoders for Sentence Classification
• Training Production Language Models without Memorizing User Data
• Echo State Neural Machine Translation
• Pufferfish: Communication-efficient Models At No Extra Cost