SoK: Training Machine Learning Models over Multiple Sources with Privacy Preservation

Abstract

Nowadays, gathering high-quality training data from multiple data sources with privacy preservation is a crucial challenge to training high-performance machine learning models. The potential solutions could break the barriers among isolated data corpus, and consequently enlarge the range of data available for processing. To this end, both academic researchers and industrial vendors are recently strongly motivated to propose two main-stream folders of solutions mainly based on software constructions: 1) Secure Multi-party Learning (MPL for short); and 2) Federated Learning (FL for short). The above two technical folders have their advantages and limitations when we evaluate them according to the following five criteria: security, efficiency, data distribution, the accuracy of trained models, and application scenarios. Motivated to demonstrate the research progress and discuss the insights on the future directions, we thoroughly investigate these protocols and frameworks of both MPL and FL. At first, we define the problem of Training machine learning Models over Multiple data sources with Privacy Preservation (TMMPP for short). Then, we compare the recent studies of TMMPP from the aspects of the technical routes, the number of parties supported, data partitioning, threat model, and machine learning models supported, to show their advantages and limitations. Next, we investigate and evaluate five popular FL platforms. Finally, we discuss the potential directions to resolve the problem of TMMPP in the future.

arXiv comment19pages, 4 figures

References in corpus (22)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning
• Learning Differentially Private Recurrent Language Models
• Differentially Private Federated Learning: A Client Level Perspective
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• A generic framework for privacy preserving deep learning
• CryptoDL: Deep Neural Networks over Encrypted Data
• Protection Against Reconstruction and Its Applications in Private Federated Learning
• SecureBoost: A Lossless Federated Learning Framework
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Variational Federated Multi-Task Learning
• A Communication Efficient Collaborative Learning Framework for Distributed Features
• Local Differential Privacy based Federated Learning for Internet of Things
• SEALion: a Framework for Neural Network Inference on Encrypted Data
• Differentially-Private "Draw and Discard" Machine Learning
• Differentially Private Meta-Learning
• Quantification of the Leakage in Federated Learning