Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning

Abstract

Recently, deep learning approach, especially deep Convolutional Neural Networks (ConvNets), have achieved overwhelming accuracy with fast processing speed for image classification. Incorporating temporal structure with deep ConvNets for video representation becomes a fundamental problem for video content analysis. In this paper, we propose a new approach, namely Hierarchical Recurrent Neural Encoder (HRNE), to exploit temporal information of videos. Compared to recent video representation inference approaches, this paper makes the following three contributions. First, our HRNE is able to efficiently exploit video temporal structure in a longer range by reducing the length of input information flow, and compositing multiple consecutive inputs at a higher level. Second, computation operations are significantly lessened while attaining more non-linearity. Third, HRNE is able to uncover temporal transitions between frame chunks with different granularities, i.e., it can model the temporal transitions between frames as well as the transitions between segments. We apply the new method to video captioning where temporal information plays a crucial role. Experiments demonstrate that our method outperforms the state-of-the-art on video captioning benchmarks. Notably, even using a single network with only RGB stream as input, HRNE beats all the recent systems which combine multiple inputs, such as RGB ConvNet plus 3D ConvNet.

References in corpus (3)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Theano: new features and speed improvements

Cited by in corpus (14)

• Delving Deeper into Convolutional Networks for Learning Video Representations
• Weakly Supervised Dense Video Captioning
• Less Is More: Picking Informative Frames for Video Captioning
• Spatio-Temporal Attention Models for Grounded Video Captioning
• Video Captioning via Hierarchical Reinforcement Learning
• An Attempt towards Interpretable Audio-Visual Video Captioning
• Video Captioning with Transferred Semantic Attributes
• Multimodal Memory Modelling for Video Captioning
• Efficient Video Classification Using Fewer Frames
• MTLE: A Multitask Learning Encoder of Visual Feature Representations for Video and Movie Description
• Few-Shot Object Recognition from Machine-Labeled Web Images
• Generating Video Descriptions with Topic Guidance
• Middle-Out Decoding
• Bidirectional Multirate Reconstruction for Temporal Modeling in Videos