Detailed 2D-3D Joint Representation for Human-Object Interaction

Abstract

Human-Object Interaction (HOI) detection lies at the core of action understanding. Besides 2D information such as human/object appearance and locations, 3D pose is also usually utilized in HOI learning since its view-independence. However, rough 3D body joints just carry sparse body information and are not sufficient to understand complex interactions. Thus, we need detailed 3D body shape to go further. Meanwhile, the interacted object in 3D is also not fully studied in HOI learning. In light of these, we propose a detailed 2D-3D joint representation learning method. First, we utilize the single-view human body capture method to obtain detailed 3D body, face and hand shapes. Next, we estimate the 3D object location and size with reference to the 2D human-object spatial configuration and object category priors. Finally, a joint learning framework and cross-modal consistency tasks are proposed to learn the joint HOI representation. To better evaluate the 2D ambiguity processing capacity of models, we propose a new benchmark named Ambiguous-HOI consisting of hard ambiguous images. Extensive experiments in large-scale HOI benchmark and Ambiguous-HOI show impressive effectiveness of our method. Code and data are available at https://github.com/DirtyHarryLYL/DJ-RN.

Accepted to CVPR 2020, supplementary materials included, code available:https://github.com/DirtyHarryLYL/DJ-RN

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Visual Semantic Role Labeling
• Detecting and Recognizing Human-Object Interactions
• HAKE: Human Activity Knowledge Engine
• Pose-aware Multi-level Feature Network for Human Object Interaction Detection
• Pointwise Rotation-Invariant Network with Adaptive Sampling and 3D Spherical Voxel Convolution
• Beyond Holistic Object Recognition: Enriching Image Understanding with Part States

Cited by in corpus (3)

• Transferable Interactiveness Knowledge for Human-Object Interaction Detection
• HOI Analysis: Integrating and Decomposing Human-Object Interaction
• Learning Single/Multi-Attribute of Object with Symmetry and Group