Tackling Dynamics in Federated Incremental Learning with Variational Embedding Rehearsal
arXiv:2110.09695
Abstract
Federated Learning is a fast growing area of ML where the training datasets are extremely distributed, all while dynamically changing over time. Models need to be trained on clients' devices without any guarantees for either homogeneity or stationarity of the local private data. The need for continual training has also risen, due to the ever-increasing production of in-task data. However, pursuing both directions at the same time is challenging, since client data privacy is a major constraint, especially for rehearsal methods. Herein, we propose a novel algorithm to address the incremental learning process in an FL scenario, based on realistic client enrollment scenarios where clients can drop in or out dynamically. We first propose using deep Variational Embeddings that secure the privacy of the client data. Second, we propose a server-side training method that enables a model to rehearse the previously learnt knowledge. Finally, we investigate the performance of federated incremental learning in dynamic client enrollment scenarios. The proposed method shows parity with offline training on domain-incremental learning, addressing challenges in both the dynamic enrollment of clients and the domain shifting of client data.
References in corpus (10)
- Auto-Encoding Variational Bayes
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- A continual learning survey: Defying forgetting in classification tasks
- A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
- Federated Optimization:Distributed Optimization Beyond the Datacenter
- Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
- A Field Guide to Federated Optimization
- Concept drift detection and adaptation for federated and continual learning
- Inverting face embeddings with convolutional neural networks
- Towards Causal Federated Learning For Enhanced Robustness and Privacy