RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning
arXiv:2007.06271
Abstract
Research on continual learning has led to a variety of approaches to mitigating catastrophic forgetting in feed-forward classification networks. Until now surprisingly little attention has been focused on continual learning of recurrent models applied to problems like image captioning. In this paper we take a systematic look at continual learning of LSTM-based models for image captioning. We propose an attention-based approach that explicitly accommodates the transient nature of vocabularies in continual image captioning tasks -- i.e. that task vocabularies are not disjoint. We call our method Recurrent Attention to Transient Tasks (RATT), and also show how to adapt continual learning approaches based on weight egularization and knowledge distillation to recurrent continual learning problems. We apply our approaches to incremental image captioning problem on two new continual learning benchmarks we define using the MS-COCO and Flickr30 datasets. Our results demonstrate that RATT is able to sequentially learn five captioning tasks while incurring no forgetting of previously learned ones.
arXiv comment9 pages, 4 figures, 8 supplementary pages, 12 supplementary images, to be published in NeurIPS 2020
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- A continual learning survey: Defying forgetting in classification tasks
- Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
- Continual Learning Through Synaptic Intelligence
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- Gradient Episodic Memory for Continual Learning
- Progress & Compress: A scalable framework for continual learning
- Continual Learning with Deep Generative Replay
- Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
- Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting
- End-to-End Incremental Learning
- Semantic Drift Compensation for Class-Incremental Learning