Federated Continual Learning with Weighted Inter-client Transfer

Abstract

There has been a surge of interest in continual learning and federated learning, both of which are important in deep neural networks in real-world scenarios. Yet little research has been done regarding the scenario where each client learns on a sequence of tasks from a private local data stream. This problem of federated continual learning poses new challenges to continual learning, such as utilizing knowledge from other clients, while preventing interference from irrelevant knowledge. To resolve these issues, we propose a novel federated continual learning framework, Federated Weighted Inter-client Transfer (FedWeIT), which decomposes the network weights into global federated parameters and sparse task-specific parameters, and each client receives selective knowledge from other clients by taking a weighted combination of their task-specific parameters. FedWeIT minimizes interference between incompatible tasks, and also allows positive knowledge transfer across clients during learning. We validate our FedWeIT against existing federated learning and continual learning methods under varying degrees of task similarity across clients, and our model significantly outperforms them with a large reduction in the communication cost. Code is available at https://github.com/wyjeong/FedWeIT

ICML 2021

References in corpus (9)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Efficient Lifelong Learning with A-GEM
• Personalized Federated Learning: A Meta-Learning Approach
• Adaptive Personalized Federated Learning
• Learning Task Grouping and Overlap in Multi-task Learning
• Bayesian Nonparametric Federated Learning of Neural Networks
• Overcoming Forgetting in Federated Learning on Non-IID Data
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Federated Learning with Matched Averaging

Cited by in corpus (8)

• Robust Federated Learning with Noisy Labels
• Concept drift detection and adaptation for federated and continual learning
• End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things
• A distillation-based approach integrating continual learning and federated learning for pervasive services
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
• Adversarial training in communication constrained federated learning
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Federated Reconnaissance: Efficient, Distributed, Class-Incremental Learning