Learning Discriminative Motion Features Through Detection

Abstract

Despite huge success in the image domain, modern detection models such as Faster R-CNN have not been used nearly as much for video analysis. This is arguably due to the fact that detection models are designed to operate on single frames and as a result do not have a mechanism for learning motion representations directly from video. We propose a learning procedure that allows detection models such as Faster R-CNN to learn motion features directly from the RGB video data while being optimized with respect to a pose estimation task. Given a pair of video frames---Frame A and Frame B---we force our model to predict human pose in Frame A using the features from Frame B. We do so by leveraging deformable convolutions across space and time. Our network learns to spatially sample features from Frame B in order to maximize pose detection accuracy in Frame A. This naturally encourages our network to learn motion offsets encoding the spatial correspondences between the two frames. We refer to these motion offsets as DiMoFs (Discriminative Motion Features). In our experiments we show that our training scheme helps learn effective motion cues, which can be used to estimate and localize salient human motion. Furthermore, we demonstrate that as a byproduct, our model also learns features that lead to improved pose detection in still-images, and better keypoint tracking. Finally, we show how to leverage our learned model for the tasks of spatiotemporal action localization and fine-grained action recognition.

References in corpus (23)

• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Understanding Neural Networks Through Deep Visualization
• Focal Loss for Dense Object Detection
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Deep Clustering for Unsupervised Learning of Visual Features
• Feature Pyramid Networks for Object Detection
• Deformable Convolutional Networks
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Xception: Deep Learning with Depthwise Separable Convolutions
• Convolutional Pose Machines
• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• Attentional Pooling for Action Recognition
• Flow-Guided Feature Aggregation for Video Object Detection
• A Better Baseline for AVA
• Temporal Relational Reasoning in Videos
• Object Detection in Video with Spatiotemporal Sampling Networks
• Actions ~ Transformations
• Actor-Centric Relation Network
• Convolutional Random Walk Networks for Semantic Image Segmentation
• Video Object Detection with an Aligned Spatial-Temporal Memory

Cited by in corpus (6)

• Grouped Spatial-Temporal Aggregation for Efficient Action Recognition
• Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning
• Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition
• Temporal Query Networks for Fine-grained Video Understanding
• SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos
• Video Modeling with Correlation Networks