Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation

Abstract

Federated learning obtains a central model on the server by aggregating models trained locally on clients. As a result, federated learning does not require clients to upload their data to the server, thereby preserving the data privacy of the clients. One challenge in federated learning is to reduce the client-server communication since the end devices typically have very limited communication bandwidth. This paper presents an enhanced federated learning technique by proposing a synchronous learning strategy on the clients and a temporally weighted aggregation of the local models on the server. In the asynchronous learning strategy, different layers of the deep neural networks are categorized into shallow and deeps layers and the parameters of the deep layers are updated less frequently than those of the shallow layers. Furthermore, a temporally weighted aggregation strategy is introduced on the server to make use of the previously trained local models, thereby enhancing the accuracy and convergence of the central model. The proposed algorithm is empirically on two datasets with different deep neural networks. Our results demonstrate that the proposed asynchronous federated deep learning outperforms the baseline algorithm both in terms of communication cost and model accuracy.

References in corpus (10)

• LSTM: A Search Space Odyssey
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• How transferable are features in deep neural networks?
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• One weird trick for parallelizing convolutional neural networks
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• Differentially Private Federated Learning: A Client Level Perspective
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• AIDE: Fast and Communication Efficient Distributed Optimization

Cited by in corpus (53)

• Towards Personalized Federated Learning
• From Distributed Machine Learning to Federated Learning: A Survey
• Federated Learning for Healthcare Domain - Pipeline, Applications and Challenges
• A Systematic Literature Review on Federated Machine Learning: From A Software Engineering Perspective
• Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning
• Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data
• Resource-Efficient Federated Learning
• Distilled One-Shot Federated Learning
• An Efficient and Reliable Asynchronous Federated Learning Scheme for Smart Public Transportation
• Free-riders in Federated Learning: Attacks and Defenses
• Federated Learning for Healthcare Informatics
• Communication Efficiency in Federated Learning: Achievements and Challenges
• CSAFL: A Clustered Semi-Asynchronous Federated Learning Framework
• Asynchronous Federated Learning on Heterogeneous Devices: A Survey
• Towards Communication-efficient and Attack-Resistant Federated Edge Learning for Industrial Internet of Things
• One-shot Federated Learning without Server-side Training
• Fairness and Accuracy in Federated Learning
• Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
• FedHe: Heterogeneous Models and Communication-Efficient Federated Learning
• Federated Continual Learning with Weighted Inter-client Transfer
• A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Things
• FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers
• A Lagrangian Dual-based Theory-guided Deep Neural Network
• A Systematic Literature Review on Federated Learning: From A Model Quality Perspective
• Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
• Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise
• Real-time Federated Evolutionary Neural Architecture Search
• Federated Learning for Energy Constrained IoT devices: A systematic mapping study
• Privacy-Preserving Federated Learning for UAV-Enabled Networks: Learning-Based Joint Scheduling and Resource Management
• Ternary Compression for Communication-Efficient Federated Learning
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• Asynchronous Online Federated Learning with Reduced Communication Requirements
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Federated Face Recognition
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• Fast-Convergent Federated Learning with Adaptive Weighting
• EventGraD: Event-Triggered Communication in Parallel Machine Learning
• Federated Learning on Non-IID Data: A Survey
• CovidSens: A Vision on Reliable Social Sensing for COVID-19
• Network-Aware Optimization of Distributed Learning for Fog Computing
• Distributed Additive Encryption and Quantization for Privacy Preserving Federated Deep Learning
• Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation
• Federated Recommender System with Data Valuation for E-commerce Platform
• FedSup: A Communication-Efficient Federated Learning Fatigue Driving Behaviors Supervision Framework
• State-of-the-art Techniques in Deep Edge Intelligence
• Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes
• FedSA-GCL: A Semi-Asynchronous Federated Graph Learning Framework with Personalized Aggregation and Cluster-Aware Broadcasting
• Gradual Federated Learning with Simulated Annealing
• Towards Explainable Multi-Party Learning: A Contrastive Knowledge Sharing Framework
• Self-Organizing Map assisted Deep Autoencoding Gaussian Mixture Model for Intrusion Detection
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations
• Distributed Learning and its Application for Time-Series Prediction
• HADFL: Heterogeneity-aware Decentralized Federated Learning Framework