DAP3D-Net: Where, What and How Actions Occur in Videos?

Abstract

Action parsing in videos with complex scenes is an interesting but challenging task in computer vision. In this paper, we propose a generic 3D convolutional neural network in a multi-task learning manner for effective Deep Action Parsing (DAP3D-Net) in videos. Particularly, in the training phase, action localization, classification and attributes learning can be jointly optimized on our appearancemotion data via DAP3D-Net. For an upcoming test video, we can describe each individual action in the video simultaneously as: Where the action occurs, What the action is and How the action is performed. To well demonstrate the effectiveness of the proposed DAP3D-Net, we also contribute a new Numerous-category Aligned Synthetic Action dataset, i.e., NASA, which consists of 200; 000 action clips of more than 300 categories and with 33 pre-defined action attributes in two hierarchical levels (i.e., low-level attributes of basic body part movements and high-level attributes related to action motion). We learn DAP3D-Net using the NASA dataset and then evaluate it on our collected Human Action Understanding (HAU) dataset. Experimental results show that our approach can accurately localize, categorize and describe multiple actions in realistic videos.

References in corpus (6)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Learning to track for spatio-temporal action localization

Cited by in corpus (5)

• Depth2Action: Exploring Embedded Depth for Large-Scale Action Recognition
• DAVE: A Unified Framework for Fast Vehicle Detection and Annotation
• Efficient Action Detection in Untrimmed Videos via Multi-Task Learning
• Exploring Temporal Information for Improved Video Understanding
• Learning Conditional Random Fields with Augmented Observations for Partially Observed Action Recognition