Lifelong Teacher-Student Network Learning
arXiv:2107.04689 · doi:10.1109/TPAMI.2021.3092677
Abstract
A unique cognitive capability of humans consists in their ability to acquire new knowledge and skills from a sequence of experiences. Meanwhile, artificial intelligence systems are good at learning only the last given task without being able to remember the databases learnt in the past. We propose a novel lifelong learning methodology by employing a Teacher-Student network framework. While the Student module is trained with a new given database, the Teacher module would remind the Student about the information learnt in the past. The Teacher, implemented by a Generative Adversarial Network (GAN), is trained to preserve and replay past knowledge corresponding to the probabilistic representations of previously learn databases. Meanwhile, the Student module is implemented by a Variational Autoencoder (VAE) which infers its latent variable representation from both the output of the Teacher module as well as from the newly available database. Moreover, the Student module is trained to capture both continuous and discrete underlying data representations across different domains. The proposed lifelong learning framework is applied in supervised, semi-supervised and unsupervised training. The code is available~: \url{https://github.com/dtuzi123/Lifelong-Teacher-Student-Network-Learning}
18 pages, 18 figures. in IEEE Transactions on Pattern Analysis and Machine Intelligence
References in corpus (31)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Distilling the Knowledge in a Neural Network
- Overcoming catastrophic forgetting in neural networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Categorical Reparameterization with Gumbel-Softmax
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- FitNets: Hints for Thin Deep Nets
- Semi-Supervised Learning with Deep Generative Models
- Progressive Neural Networks
- Continual Learning Through Synaptic Intelligence
- Disentangling by Factorising
- Three scenarios for continual learning
- VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
- Domain Adaptation: Learning Bounds and Algorithms
- Understanding disentangling in -VAE
- An introduction to domain adaptation and transfer learning
- Less-forgetting Learning in Deep Neural Networks
- Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
- Learning Disentangled Representations with Semi-Supervised Deep Generative Models
- Lifelong Generative Modeling
- AdaNet: Adaptive Structural Learning of Artificial Neural Networks
- Continual Learning in Generative Adversarial Nets
- DIVA: Domain Invariant Variational Autoencoders
- Continual Unsupervised Representation Learning
- Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies
- Learning Latent Subspaces in Variational Autoencoders
- Adversarial Symmetric Variational Autoencoder
- Auto-Encoding Total Correlation Explanation
- A survey on domain adaptation theory: learning bounds and theoretical guarantees
- Adversarial Information Factorization