Action Machine: Rethinking Action Recognition in Trimmed Videos

Abstract

Existing methods in video action recognition mostly do not distinguish human body from the environment and easily overfit the scenes and objects. In this work, we present a conceptually simple, general and high-performance framework for action recognition in trimmed videos, aiming at person-centric modeling. The method, called Action Machine, takes as inputs the videos cropped by person bounding boxes. It extends the Inflated 3D ConvNet (I3D) by adding a branch for human pose estimation and a 2D CNN for pose-based action recognition, being fast to train and test. Action Machine can benefit from the multi-task training of action recognition and pose estimation, the fusion of predictions from RGB images and poses. On NTU RGB-D, Action Machine achieves the state-of-the-art performance with top-1 accuracies of 97.2% and 94.3% on cross-view and cross-subject respectively. Action Machine also achieves competitive performance on another three smaller action recognition datasets: Northwestern UCLA Multiview Action3D, MSR Daily Activity3D and UTD-MHAD. Code will be made available.

References in corpus (22)

• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Stacked Hourglass Networks for Human Pose Estimation
• Deformable Convolutional Networks
• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
• A Short Note about Kinetics-600
• Convolutional Pose Machines
• Learning Deep Features for Discriminative Localization
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Non-local Neural Networks
• Cascaded Pyramid Network for Multi-Person Pose Estimation
• BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
• Cross-view Action Modeling, Learning and Recognition
• View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data
• The ActivityNet Large-Scale Activity Recognition Challenge 2018 Summary
• ActivityNet Challenge 2017 Summary
• View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition
• Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos
• End-to-end Video-level Representation Learning for Action Recognition
• Multimodal Multipart Learning for Action Recognition in Depth Videos

Cited by in corpus (6)

• From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
• Pose is all you need: The pose only group activity recognition system (POGARS)
• Multi-Stage HRNet: Multiple Stage High-Resolution Network for Human Pose Estimation
• Human Following for Wheeled Robot with Monocular Pan-tilt Camera
• Motion Control on Bionic Eyes: A Comprehensive Review
• High Performance Visual Object Tracking with Unified Convolutional Networks