Detailed comparison of communication efficiency of split learning and federated learning

Abstract

We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and juxtapose the two methods under various real-life scenarios. We consider settings of small and large number of clients as well as small models (1M - 6M parameters), large models (10M - 200M parameters) and very large models (1 Billion-100 Billion parameters). We show that increasing number of clients or increasing model size favors split learning setup over the federated while increasing the number of data samples while keeping the number of clients or model size low makes federated learning more communication efficient.

References in corpus (2)

• Federated Learning: Strategies for Improving Communication Efficiency
• Split learning for health: Distributed deep learning without sharing raw patient data

Cited by in corpus (10)

• FedML: A Research Library and Benchmark for Federated Machine Learning
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• Split Learning for collaborative deep learning in healthcare
• A Federated Learning Framework for Healthcare IoT devices
• End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things
• Communication and Computation Reduction for Split Learning using Asynchronous Training
• FedV: Privacy-Preserving Federated Learning over Vertically Partitioned Data
• Splitfed learning without client-side synchronization: Analyzing client-side split network portion size to overall performance
• Towards Representation Identical Privacy-Preserving Graph Neural Network via Split Learning
• Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modeling