Federated Continual Learning via Knowledge Fusion: A Survey

Abstract

Data privacy and silos are nontrivial and greatly challenging in many real-world applications. Federated learning is a decentralized approach to training models across multiple local clients without the exchange of raw data from client devices to global servers. However, existing works focus on a static data environment and ignore continual learning from streaming data with incremental tasks. Federated Continual Learning (FCL) is an emerging paradigm to address model learning in both federated and continual learning environments. The key objective of FCL is to fuse heterogeneous knowledge from different clients and retain knowledge of previous tasks while learning on new ones. In this work, we delineate federated learning and continual learning first and then discuss their integration, i.e., FCL, and particular FCL via knowledge fusion. In summary, our motivations are four-fold: we (1) raise a fundamental problem called ''spatial-temporal catastrophic forgetting'' and evaluate its impact on the performance using a well-known method called federated averaging (FedAvg), (2) integrate most of the existing FCL methods into two generic frameworks, namely synchronous FCL and asynchronous FCL, (3) categorize a large number of methods according to the mechanism involved in knowledge fusion, and finally (4) showcase an outlook on the future work of FCL.

20 pages

References in corpus (38)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Prototypical Networks for Few-shot Learning
• Squeeze-and-Excitation Networks
• A continual learning survey: Defying forgetting in classification tasks
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• Progressive Neural Networks
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• How To Backdoor Federated Learning
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Federated Machine Learning: Concept and Applications
• Three scenarios for continual learning
• Federated Learning with Personalization Layers
• iDLG: Improved Deep Leakage from Gradients
• Experience Replay for Continual Learning
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
• Progress & Compress: A scalable framework for continual learning
• Threats to Federated Learning: A Survey
• Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
• Robust Federated Learning in a Heterogeneous Environment
• Overcoming Forgetting in Federated Learning on Non-IID Data
• Variational Federated Multi-Task Learning
• Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks
• FedAvg with Fine Tuning: Local Updates Lead to Representation Learning
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning
• Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentation
• A distillation-based approach integrating continual learning and federated learning for pervasive services
• Continual Learning of Natural Language Processing Tasks: A Survey
• A Theoretical Study on Solving Continual Learning
• Responsible Facial Recognition and Beyond
• Continual Horizontal Federated Learning for Heterogeneous Data
• Continual Learning for Peer-to-Peer Federated Learning: A Study on Automated Brain Metastasis Identification
• Federated and continual learning for classification tasks in a society of devices
• Optimizing Privacy, Utility and Efficiency in Constrained Multi-Objective Federated Learning
• Federated Learning and catastrophic forgetting in pervasive computing: demonstration in HAR domain
• ReLoop: A Self-Correction Continual Learning Loop for Recommender Systems
• Addressing Client Drift in Federated Continual Learning with Adaptive Optimization
• Federated Continual Learning to Detect Accounting Anomalies in Financial Auditing

Cited by in corpus (6)

• Federated Continual Learning: Concepts, Challenges, and Solutions
• Personalized Federated Continual Learning via Multi-granularity Prompt
• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• One-Shot Federated Clustering of Non-Independent Completely Distributed Data
• ErrorEraser: Unlearning Data Bias for Improved Continual Learning
• Federated continual learning: A comprehensive survey on lifelong and privacy-preserving learning over distributed and non-stationary data