Overcoming Forgetting in Federated Learning on Non-IID Data

Abstract

We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning. Building on an analogy with Lifelong Learning, we adapt a solution for catastrophic forgetting to Federated Learning. We add a penalty term to the loss function, compelling all local models to converge to a shared optimum. We show that this can be done efficiently for communication (adding no further privacy risks), scaling with the number of nodes in the distributed setting. Our experiments show that this method is superior to competing ones for image recognition on the MNIST dataset.

arXiv commentAccepted to NeurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality

References in corpus (8)

• Overcoming catastrophic forgetting in neural networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• Federated Optimization in Heterogeneous Networks
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• Practical Secure Aggregation for Federated Learning on User-Held Data
• AIDE: Fast and Communication Efficient Distributed Optimization

Cited by in corpus (24)

• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Federated Continual Learning via Knowledge Fusion: A Survey
• Robust Federated Learning with Noisy Labels
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Distilled One-Shot Federated Learning
• A Bayesian Federated Learning Framework with Online Laplace Approximation
• Fairness and Accuracy in Federated Learning
• Federated Continual Learning with Weighted Inter-client Transfer
• FedCD: Improving Performance in non-IID Federated Learning
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
• Federated brain tumor segmentation: an extensive benchmark
• An On-Device Federated Learning Approach for Cooperative Model Update between Edge Devices
• Adversarial training in communication constrained federated learning
• TinyFedTL: Federated Transfer Learning on Tiny Devices
• BlockFLA: Accountable Federated Learning via Hybrid Blockchain Architecture
• Federated Learning on Non-IID Data: A Survey
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• Towards Fair Federated Learning with Zero-Shot Data Augmentation
• Explainable Semantic Federated Learning Enabled Industrial Edge Network for Fire Surveillance
• Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor
• Cross-Gradient Aggregation for Decentralized Learning from Non-IID data
• Demystifying the Effects of Non-Independence in Federated Learning
• FedNNNN: Norm-Normalized Neural Network Aggregation for Fast and Accurate Federated Learning
• Towards More Efficient Federated Learning with Better Optimization Objects