Bidirectional Multirate Reconstruction for Temporal Modeling in Videos

Abstract

Despite the recent success of neural networks in image feature learning, a major problem in the video domain is the lack of sufficient labeled data for learning to model temporal information. In this paper, we propose an unsupervised temporal modeling method that learns from untrimmed videos. The speed of motion varies constantly, e.g., a man may run quickly or slowly. We therefore train a Multirate Visual Recurrent Model (MVRM) by encoding frames of a clip with different intervals. This learning process makes the learned model more capable of dealing with motion speed variance. Given a clip sampled from a video, we use its past and future neighboring clips as the temporal context, and reconstruct the two temporal transitions, i.e., present$\to$past transition and present$\to$future transition, reflecting the temporal information in different views. The proposed method exploits the two transitions simultaneously by incorporating a bidirectional reconstruction which consists of a backward reconstruction and a forward reconstruction. We apply the proposed method to two challenging video tasks, i.e., complex event detection and video captioning, in which it achieves state-of-the-art performance. Notably, our method generates the best single feature for event detection with a relative improvement of 10.4% on the MEDTest-13 dataset and achieves the best performance in video captioning across all evaluation metrics on the YouTube2Text dataset.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• TensorFlow: A system for large-scale machine learning
• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Spatio-temporal video autoencoder with differentiable memory
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Delving Deeper into Convolutional Networks for Learning Video Representations
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
• CIDEr: Consensus-based Image Description Evaluation
• Uncovering Temporal Context for Video Question and Answering
• Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
• Jointly Modeling Embedding and Translation to Bridge Video and Language
• A Discriminative CNN Video Representation for Event Detection
• Beyond Gaussian Pyramid: Multi-skip Feature Stacking for Action Recognition