Memory Warps for Learning Long-Term Online Video Representations

Abstract

This paper proposes a novel memory-based online video representation that is efficient, accurate and predictive. This is in contrast to prior works that often rely on computationally heavy 3D convolutions, ignore actual motion when aligning features over time, or operate in an off-line mode to utilize future frames. In particular, our memory (i) holds the feature representation, (ii) is spatially warped over time to compensate for observer and scene motions, (iii) can carry long-term information, and (iv) enables predicting feature representations in future frames. By exploring a variant that operates at multiple temporal scales, we efficiently learn across even longer time horizons. We apply our online framework to object detection in videos, obtaining a large 2.3 times speed-up and losing only 0.9% mAP on ImageNet-VID dataset, compared to prior works that even use future frames. Finally, we demonstrate the predictive property of our representation in two novel detection setups, where features are propagated over time to (i) significantly enhance a real-time detector by more than 10% mAP in a multi-threaded online setup and to (ii) anticipate objects in future frames.

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Unsupervised Learning of Video Representations using LSTMs
• Focal Loss for Dense Object Detection
• YouTube-8M: A Large-Scale Video Classification Benchmark
• FlowNet: Learning Optical Flow with Convolutional Networks
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Unsupervised Learning of Visual Representations using Videos
• Describing Videos by Exploiting Temporal Structure

Cited by in corpus (5)

• FSDR: Frequency Space Domain Randomization for Domain Generalization
• Contextual-Relation Consistent Domain Adaptation for Semantic Segmentation
• Memory-Augmented Temporal Dynamic Learning for Action Recognition
• Understanding Road Layout from Videos as a Whole
• Functionally Modular and Interpretable Temporal Filtering for Robust Segmentation