Decentralized Learning of Generative Adversarial Networks from Non-iid Data

Abstract

This work addresses a new problem that learns generative adversarial networks (GANs) from multiple data collections that are each i) owned separately by different clients and ii) drawn from a non-identical distribution that comprises different classes. Given such non-iid data as input, we aim to learn a distribution involving all the classes input data can belong to, while keeping the data decentralized in each client storage. Our key contribution to this end is a new decentralized approach for learning GANs from non-iid data called Forgiver-First Update (F2U), which a) asks clients to train an individual discriminator with their own data and b) updates a generator to fool the most `forgiving' discriminators who deem generated samples as the most real. Our theoretical analysis proves that this updating strategy allows the decentralized GAN to achieve a generator's distribution with all the input classes as its global optimum based on f-divergence minimization. Moreover, we propose a relaxed version of F2U called Forgiver-First Aggregation (F2A) that performs well in practice, which adaptively aggregates the discriminators while emphasizing forgiving ones. Our empirical evaluations with image generation tasks demonstrated the effectiveness of our approach over state-of-the-art decentralized learning methods.

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Conditional Generative Adversarial Nets
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Federated Learning: Strategies for Improving Communication Efficiency
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• YouTube-8M: A Large-Scale Video Classification Benchmark
• How To Backdoor Federated Learning
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Adversarial Discriminative Domain Adaptation
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets
• Generative Multi-Adversarial Networks

Cited by in corpus (7)

• FedGAN: Federated Generative Adversarial Networks for Distributed Data
• Generative Adversarial Networks: A Survey Towards Private and Secure Applications
• Multi-hop Federated Private Data Augmentation with Sample Compression
• Brainstorming Generative Adversarial Networks (BGANs): Towards Multi-Agent Generative Models with Distributed Private Datasets
• Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent
• A New Distributed Method for Training Generative Adversarial Networks
• Continual Learning of Generative Models with Limited Data: From Wasserstein-1 Barycenter to Adaptive Coalescence