HAKE: Human Activity Knowledge Engine

Abstract

Human activity understanding is crucial for building automatic intelligent system. With the help of deep learning, activity understanding has made huge progress recently. But some challenges such as imbalanced data distribution, action ambiguity, complex visual patterns still remain. To address these and promote the activity understanding, we build a large-scale Human Activity Knowledge Engine (HAKE) based on the human body part states. Upon existing activity datasets, we annotate the part states of all the active persons in all images, thus establish the relationship between instance activity and body part states. Furthermore, we propose a HAKE based part state recognition model with a knowledge extractor named Activity2Vec and a corresponding part state based reasoning network. With HAKE, our method can alleviate the learning difficulty brought by the long-tail data distribution, and bring in interpretability. Now our HAKE has more than 7 M+ part state annotations and is still under construction. We first validate our approach on a part of HAKE in this preliminary paper, where we show 7.2 mAP performance improvement on Human-Object Interaction recognition, and 12.38 mAP improvement on the one-shot subsets.

Work in progress. Project website: http://hake-mvig.cn

References in corpus (8)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Visual Semantic Role Labeling
• iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection
• Transferable Interactiveness Knowledge for Human-Object Interaction Detection
• Contextual Action Recognition with R*CNN
• Care about you: towards large-scale human-centric visual relationship detection
• Beyond Holistic Object Recognition: Enriching Image Understanding with Part States

Cited by in corpus (8)

• Multi-Modal Knowledge Graph Construction and Application: A Survey
• Detailed 2D-3D Joint Representation for Human-Object Interaction
• Recursive Social Behavior Graph for Trajectory Prediction
• KeypointNet: A Large-scale 3D Keypoint Dataset Aggregated from Numerous Human Annotations
• TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model
• Is Object Detection Necessary for Human-Object Interaction Recognition?
• A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
• Symmetry and Group in Attribute-Object Compositions