Reducing Communication for Split Learning by Randomized Top-k Sparsification

Abstract

Split learning is a simple solution for Vertical Federated Learning (VFL), which has drawn substantial attention in both research and application due to its simplicity and efficiency. However, communication efficiency is still a crucial issue for split learning. In this paper, we investigate multiple communication reduction methods for split learning, including cut layer size reduction, top-k sparsification, quantization, and L1 regularization. Through analysis of the cut layer size reduction and top-k sparsification, we further propose randomized top-k sparsification, to make the model generalize and converge better. This is done by selecting top-k elements with a large probability while also having a small probability to select non-top-k elements. Empirical results show that compared with other communication-reduction methods, our proposed randomized top-k sparsification achieves a better model performance under the same compression level.

arXiv commentAccepted by IJCAI 2023

References in corpus (10)

• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Exploring Generalization in Deep Learning
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Norm-Based Capacity Control in Neural Networks
• Accelerated Federated Learning with Decoupled Adaptive Optimization
• Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data
• Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates