Cross-view Action Modeling, Learning and Recognition

Abstract

Existing methods on video-based action recognition are generally view-dependent, i.e., performing recognition from the same views seen in the training data. We present a novel multiview spatio-temporal AND-OR graph (MST-AOG) representation for cross-view action recognition, i.e., the recognition is performed on the video from an unknown and unseen view. As a compositional model, MST-AOG compactly represents the hierarchical combinatorial structures of cross-view actions by explicitly modeling the geometry, appearance and motion variations. This paper proposes effective methods to learn the structure and parameters of MST-AOG. The inference based on MST-AOG enables action recognition from novel views. The training of MST-AOG takes advantage of the 3D human skeleton data obtained from Kinect cameras to avoid annotating enormous multi-view video frames, which is error-prone and time-consuming, but the recognition does not need 3D information and is based on 2D video input. A new Multiview Action3D dataset has been created and will be released. Extensive experiments have demonstrated that this new action representation significantly improves the accuracy and robustness for cross-view action recognition on 2D videos.

CVPR 2014

Cited by in corpus (17)

• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
• PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
• Unsupervised Learning of View-invariant Action Representations
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey
• Action Machine: Rethinking Action Recognition in Trimmed Videos
• Space-Time Representation of People Based on 3D Skeletal Data: A Review
• View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition
• Skepxels: Spatio-temporal Image Representation of Human Skeleton Joints for Action Recognition
• Learning Human Pose Models from Synthesized Data for Robust RGB-D Action Recognition
• Action Recognition for Depth Video using Multi-view Dynamic Images
• Iterate & Cluster: Iterative Semi-Supervised Action Recognition
• RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks
• Viewpoint Invariant Action Recognition using RGB-D Videos
• Histogram of Oriented Principal Components for Cross-View Action Recognition
• Learning Linear Dynamical Systems with High-Order Tensor Data for Skeleton based Action Recognition
• View-invariant action recognition
• NTU-X: An Enhanced Large-scale Dataset for Improving Pose-based Recognition of Subtle Human Actions