A Communication Efficient Collaborative Learning Framework for Distributed Features

Abstract

We introduce a collaborative learning framework allowing multiple parties having different sets of attributes about the same user to jointly build models without exposing their raw data or model parameters. In particular, we propose a Federated Stochastic Block Coordinate Descent (FedBCD) algorithm, in which each party conducts multiple local updates before each communication to effectively reduce the number of communication rounds among parties, a principal bottleneck for collaborative learning problems. We analyze theoretically the impact of the number of local updates and show that when the batch size, sample size, and the local iterations are selected appropriately, within $T$ iterations, the algorithm performs $\mathcal{O}(\sqrt{T})$ communication rounds and achieves some $\mathcal{O}(1/\sqrt{T})$ accuracy (measured by the average of the gradient norm squared). The approach is supported by our empirical evaluations on a variety of tasks and datasets, demonstrating advantages over stochastic gradient descent (SGD) approaches.

This paper is published at the 2nd International Workshop on Federated Learning for Data Privacy and Confidentiality, in Conjunction with NeurIPS 2019 (FL-NeurIPS 19): https://nips.cc/Conferences/2019/ScheduleMultitrack?event=13202

References in corpus (10)

• On the Convergence of FedAvg on Non-IID Data
• Secure Federated Transfer Learning
• Federated Machine Learning: Concept and Applications
• Federated Optimization in Heterogeneous Networks
• Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
• ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates
• Distributed Coordinate Descent Method for Learning with Big Data
• SecureBoost: A Lossless Federated Learning Framework
• Learning Privately over Distributed Features: An ADMM Sharing Approach
• A distributed block coordinate descent method for training $l_1$ regularized linear classifiers

Cited by in corpus (24)

• From Distributed Machine Learning to Federated Learning: A Survey
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• VAFL: a Method of Vertical Asynchronous Federated Learning
• Multi-Participant Multi-Class Vertical Federated Learning
• Backdoor attacks and defenses in feature-partitioned collaborative learning
• Practical Vertical Federated Learning with Unsupervised Representation Learning
• Cross-Silo Federated Learning for Multi-Tier Networks with Vertical and Horizontal Data Partitioning
• FedEval: A Holistic Evaluation Framework for Federated Learning
• Self-supervised Cross-silo Federated Neural Architecture Search
• Accelerating Vertical Federated Learning
• Continual Horizontal Federated Learning for Heterogeneous Data
• Assisted Learning: A Framework for Multi-Organization Learning
• Achieving Model Fairness in Vertical Federated Learning
• A Vertical Federated Learning Framework for Horizontally Partitioned Labels
• RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data
• Hybrid Federated Learning: Algorithms and Implementation
• Federated Learning on Non-IID Data: A Survey
• SoK: Training Machine Learning Models over Multiple Sources with Privacy Preservation
• GAL: Gradient Assisted Learning for Decentralized Multi-Organization Collaborations
• FedCTR: Federated Native Ad CTR Prediction with Multi-Platform User Behavior Data
• An Information-Theoretic Analysis of The Cost of Decentralization for Learning and Inference Under Privacy Constraints
• BadVFL: Backdoor Attacks in Vertical Federated Learning
• Budgeted Online Selection of Candidate IoT Clients to Participate in Federated Learning