ContCap: A scalable framework for continual image captioning

Abstract

While advanced image captioning systems are increasingly describing images coherently and exactly, recent progress in continual learning allows deep learning models to avoid catastrophic forgetting. However, the domain where image captioning working with continual learning has not yet been explored. We define the task in which we consolidate continual learning and image captioning as continual image captioning. In this work, we propose ContCap, a framework generating captions over a series of new tasks coming, seamlessly integrating continual learning into image captioning besides addressing catastrophic forgetting. After proving forgetting in image captioning, we propose various techniques to overcome the forgetting dilemma by taking a simple fine-tuning schema as the baseline. We split MS-COCO 2014 dataset to perform experiments in class-incremental settings without revisiting dataset of previously provided tasks. Experiments show remarkable improvements in the performance on the old tasks while the figures for the new surprisingly surpass fine-tuning. Our framework also offers a scalable solution for continual image or video captioning.

8 pages

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data
• Incremental Classifier Learning with Generative Adversarial Networks

Cited by in corpus (6)

• Recent Advances of Continual Learning in Computer Vision: An Overview
• Imbalanced Continual Learning with Partitioning Reservoir Sampling
• Dissecting Catastrophic Forgetting in Continual Learning by Deep Visualization
• Visually Grounded Continual Learning of Compositional Phrases
• IROS 2019 Lifelong Robotic Vision Challenge -- Lifelong Object Recognition Report
• Explaining How Deep Neural Networks Forget by Deep Visualization