Predictive-Corrective Networks for Action Detection
arXiv:1704.03615 · doi:10.1109/CVPR.2017.223
Abstract
While deep feature learning has revolutionized techniques for static-image understanding, the same does not quite hold for video processing. Architectures and optimization techniques used for video are largely based off those for static images, potentially underutilizing rich video information. In this work, we rethink both the underlying network architecture and the stochastic learning paradigm for temporal data. To do so, we draw inspiration from classic theory on linear dynamic systems for modeling time series. By extending such models to include nonlinear mappings, we derive a series of novel recurrent neural networks that sequentially make top-down predictions about the future and then correct those predictions with bottom-up observations. Predictive-corrective networks have a number of desirable properties: (1) they can adaptively focus computation on "surprising" frames where predictions require large corrections, (2) they simplify learning in that only "residual-like" corrective terms need to be learned over time and (3) they naturally decorrelate an input data stream in a hierarchical fashion, producing a more reliable signal for learning at each layer of a network. We provide an extensive analysis of our lightweight and interpretable framework, and demonstrate that our model is competitive with the two-stream network on three challenging datasets without the need for computationally expensive optical flow.
arXiv commentAccepted to CVPR 2017. [v2]: Updated Multi-LSTM mAP on MultiTHUMOS (should be 29.7, was initially reported as 29.6). [Project URL]: http://www.achaldave.com/projects/predictive-corrective/
References in corpus (18)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Two-Stream Convolutional Networks for Action Recognition in Videos
- On the difficulty of training Recurrent Neural Networks
- Unsupervised Learning of Video Representations using LSTMs
- YouTube-8M: A Large-Scale Video Classification Benchmark
- Visualizing and Understanding Recurrent Networks
- Generating Videos with Scene Dynamics
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- Towards Good Practices for Very Deep Two-Stream ConvNets
- A Clockwork RNN
- Unsupervised Learning for Physical Interaction through Video Prediction
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
- Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Long-term Temporal Convolutions for Action Recognition
- Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
- Asynchronous Temporal Fields for Action Recognition
Cited by in corpus (13)
- Activity Graph Transformer for Temporal Action Localization
- RGB-D-based Human Motion Recognition with Deep Learning: A Survey
- Interpretable Deep Feature Propagation for Early Action Recognition
- Online recognition of unsegmented actions with hierarchical SOM architecture
- Grab: Fast and Accurate Sensor Processing for Cashier-Free Shopping
- Motion Segmentation using Frequency Domain Transformer Networks
- Progress Regression RNN for Online Spatial-Temporal Action Localization in Unconstrained Videos
- NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition
- AutoLoc: Weakly-supervised Temporal Action Localization
- Kernel Corrector LSTM
- Online Detection of Action Start in Untrimmed, Streaming Videos
- Recurrent Residual Module for Fast Inference in Videos
- Differentiable Grammars for Videos