Video Time: Properties, Encoders and Evaluation

Abstract

Time-aware encoding of frame sequences in a video is a fundamental problem in video understanding. While many attempted to model time in videos, an explicit study on quantifying video time is missing. To fill this lacuna, we aim to evaluate video time explicitly. We describe three properties of video time, namely a) temporal asymmetry, b)temporal continuity and c) temporal causality. Based on each we formulate a task able to quantify the associated property. This allows assessing the effectiveness of modern video encoders, like C3D and LSTM, in their ability to model time. Our analysis provides insights about existing encoders while also leading us to propose a new video time encoder, which is better suited for the video time recognition tasks than C3D and LSTM. We believe the proposed meta-analysis can provide a reasonable baseline to assess video time encoders on equal grounds on a set of temporal-aware tasks.

arXiv comment14 pages, BMVC 2018

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• DRAW: A Recurrent Neural Network For Image Generation
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Video (language) modeling: a baseline for generative models of natural videos
• Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
• Learning to Generate Long-term Future via Hierarchical Prediction
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Long-term Temporal Convolutions for Action Recognition
• Temporal Relational Reasoning in Videos