Generative Models for Effective ML on Private, Decentralized Datasets

Abstract

To improve real-world applications of machine learning, experienced modelers develop intuition about their datasets, their models, and how the two interact. Manual inspection of raw data - of representative samples, of outliers, of misclassifications - is an essential tool in a) identifying and fixing problems in the data, b) generating new modeling hypotheses, and c) assigning or refining human-provided labels. However, manual data inspection is problematic for privacy sensitive datasets, such as those representing the behavior of real-world individuals. Furthermore, manual data inspection is impossible in the increasingly important setting of federated learning, where raw examples are stored at the edge and the modeler may only access aggregated outputs such as metrics or model parameters. This paper demonstrates that generative models - trained using federated methods and with formal differential privacy guarantees - can be used effectively to debug many commonly occurring data issues even when the data cannot be directly inspected. We explore these methods in applications to text with differentially private federated RNNs and to images using a novel algorithm for differentially private federated GANs.

26 pages, 8 figures. Camera-ready ICLR 2020 version

References in corpus (14)

• LSTM: A Search Space Odyssey
• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Renyi Differential Privacy
• Federated Learning for Mobile Keyboard Prediction
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
• Learning Differentially Private Recurrent Language Models
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Differentially Private Generative Adversarial Network
• LEAF: A Benchmark for Federated Settings
• Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Reconstruction and Membership Inference Attacks against Generative Models

Cited by in corpus (17)

• FedML: A Research Library and Benchmark for Federated Machine Learning
• Three Approaches for Personalization with Applications to Federated Learning
• A Field Guide to Federated Optimization
• Privacy Intelligence: A Survey on Image Privacy in Online Social Networks
• Federated Learning with Only Positive Labels
• FedGAN: Federated Generative Adversarial Networks for Distributed Data
• Generative Adversarial Networks: A Survey Towards Private and Secure Applications
• Efficient Algorithms for Federated Saddle Point Optimization
• Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent
• Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift
• Tight and Robust Private Mean Estimation with Few Users
• Adapting deep generative approaches for getting synthetic data with realistic marginal distributions
• Transfer Learning with Pre-trained Conditional Generative Models
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Disentangled Unsupervised Image Translation via Restricted Information Flow
• Loosely Coupled Federated Learning Over Generative Models
• Federated Traffic Synthesizing and Classification Using Generative Adversarial Networks