GRAFFL: Gradient-free Federated Learning of a Bayesian Generative Model

Abstract

Federated learning platforms are gaining popularity. One of the major benefits is to mitigate the privacy risks as the learning of algorithms can be achieved without collecting or sharing data. While federated learning (i.e., many based on stochastic gradient algorithms) has shown great promise, there are still many challenging problems in protecting privacy, especially during the process of gradients update and exchange. This paper presents the first gradient-free federated learning framework called GRAFFL for learning a Bayesian generative model based on approximate Bayesian computation. Unlike conventional federated learning algorithms based on gradients, our framework does not require to disassemble a model (i.e., to linear components) or to perturb data (or encryption of data for aggregation) to preserve privacy. Instead, this framework uses implicit information derived from each participating institution to learn posterior distributions of parameters. The implicit information is summary statistics derived from SuffiAE that is a neural network developed in this study to create compressed and linearly separable representations thereby protecting sensitive information from leakage. As a sufficient dimensionality reduction technique, this is proved to provide sufficient summary statistics. We propose the GRAFFL-based Bayesian Gaussian mixture model to serve as a proof-of-concept of the framework. Using several datasets, we demonstrated the feasibility and usefulness of our model in terms of privacy protection and prediction performance (i.e., close to an ideal setting). The trained model as a quasi-global model can generate informative samples involving information from other institutions and enhances data analysis of each institution.

References in corpus (14)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• An overview of gradient descent optimization algorithms
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Understanding deep learning requires rethinking generalization
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Differentially Private Federated Learning: A Client Level Perspective
• A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation
• Fair Resource Allocation in Federated Learning
• Federated Tensor Factorization for Computational Phenotyping
• ABC-CDE: Towards Approximate Bayesian Computation with Complex High-Dimensional Data and Limited Simulations
• Summary Statistics in Approximate Bayesian Computation
• Deep Variational Sufficient Dimensionality Reduction