End-to-End Video Captioning with Multitask Reinforcement Learning

Abstract

Although end-to-end (E2E) learning has led to impressive progress on a variety of visual understanding tasks, it is often impeded by hardware constraints (e.g., GPU memory) and is prone to overfitting. When it comes to video captioning, one of the most challenging benchmark tasks in computer vision, those limitations of E2E learning are especially amplified by the fact that both the input videos and output captions are lengthy sequences. Indeed, state-of-the-art methods for video captioning process video frames by convolutional neural networks and generate captions by unrolling recurrent neural networks. If we connect them in an E2E manner, the resulting model is both memory-consuming and data-hungry, making it extremely hard to train. In this paper, we propose a multitask reinforcement learning approach to training an E2E video captioning model. The main idea is to mine and construct as many effective tasks (e.g., attributes, rewards, and the captions) as possible from the human captioned videos such that they can jointly regulate the search space of the E2E neural network, from which an E2E video captioning model can be found and generalized to the testing phase. To the best of our knowledge, this is the first video captioning model that is trained end-to-end from the raw video input to the caption output. Experimental results show that such a model outperforms existing ones to a large margin on two benchmark video captioning datasets.

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Frame- and Segment-Level Features and Candidate Pool Evaluation for Video Caption Generation
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Weakly Supervised Dense Video Captioning
• Fine-grained Recognition in the Wild: A Multi-Task Domain Adaptation Approach

Cited by in corpus (1)

• Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation