Overcoming Forgetting in Federated Learning on Non-IID Data

Abstract

We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning. Building on an analogy with Lifelong Learning, we adapt a solution for catastrophic forgetting to Federated Learning. We add a penalty term to the loss function, compelling all local models to converge to a shared optimum. We show that this can be done efficiently for communication (adding no further privacy risks), scaling with the number of nodes in the distributed setting. Our experiments show that this method is superior to competing ones for image recognition on the MNIST dataset.

Accepted to NeurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality

References in corpus (2)

• Practical Secure Aggregation for Federated Learning on User-Held Data
• AIDE: Fast and Communication Efficient Distributed Optimization

Cited by in corpus (9)

• Fairness and Accuracy in Federated Learning
• FedCD: Improving Performance in non-IID Federated Learning
• Adversarial training in communication constrained federated learning
• TinyFedTL: Federated Transfer Learning on Tiny Devices
• Federated Learning on Non-IID Data: A Survey
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• Demystifying the Effects of Non-Independence in Federated Learning
• Towards More Efficient Federated Learning with Better Optimization Objects
• FedNNNN: Norm-Normalized Neural Network Aggregation for Fast and Accurate Federated Learning