Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning

Abstract

Federated learning (FL) is a novel machine learning setting that enables on-device intelligence via decentralized training and federated optimization. Deep neural networks' rapid development facilitates the learning techniques for modeling complex problems and emerges into federated deep learning under the federated setting. However, the tremendous amount of model parameters burdens the communication network with a high load of transportation. This paper introduces two approaches for improving communication efficiency by dynamic sampling and top-$k$ selective masking. The former controls the fraction of selected client models dynamically, while the latter selects parameters with top-$k$ largest values of difference for federated updating. Experiments on convolutional image classification and recurrent language modeling are conducted on three public datasets to show our proposed methods' effectiveness.

arXiv commentIEEE Intelligent Systems, 2021

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Towards Federated Learning at Scale: System Design
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Secure Federated Transfer Learning
• Differentially Private Federated Learning: A Client Level Perspective
• Pointer Sentinel Mixture Models
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Agnostic Federated Learning
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Peer-to-peer Federated Learning on Graphs
• Federated Tensor Factorization for Computational Phenotyping
• Learning Private Neural Language Modeling with Attentive Aggregation
• Private Federated Learning with Domain Adaptation
• Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Results

Cited by in corpus (6)

• Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks
• FLAD: Adaptive Federated Learning for DDoS Attack Detection
• Secure Aggregation with Heterogeneous Quantization in Federated Learning
• A Federated Learning-enabled Smart Street Light Monitoring Application: Benefits and Future Challenges
• FedSAUC: A Similarity-Aware Update Control for Communication-Efficient Federated Learning in Edge Computing
• Federated Natural Language Generation for Personalized Dialogue System