LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images

Abstract

We propose an end-to-end architecture for joint 2D and 3D human pose estimation in natural images. Key to our approach is the generation and scoring of a number of pose proposals per image, which allows us to predict 2D and 3D poses of multiple people simultaneously. Hence, our approach does not require an approximate localization of the humans for initialization. Our Localization-Classification-Regression architecture, named LCR-Net, contains 3 main components: 1) the pose proposal generator that suggests candidate poses at different locations in the image; 2) a classifier that scores the different pose proposals; and 3) a regressor that refines pose proposals both in 2D and 3D. All three stages share the convolutional feature layers and are trained jointly. The final pose estimation is obtained by integrating over neighboring pose hypotheses, which is shown to improve over a standard non maximum suppression algorithm. Our method recovers full-body 2D and 3D poses, hallucinating plausible body parts when the persons are partially occluded or truncated by the image boundary. Our approach significantly outperforms the state of the art in 3D pose estimation on Human3.6M, a controlled environment. Moreover, it shows promising results on real images for both single and multi-person subsets of the MPII 2D pose benchmark and demonstrates satisfying 3D pose results even for multi-person images.

arXiv commentjournal version of the CVPR 2017 paper, accepted to appear in IEEE Trans. PAMI

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Learning from Synthetic Humans
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• Associative Embedding: End-to-End Learning for Joint Detection and Grouping
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• Human pose estimation via Convolutional Part Heatmap Regression
• Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild

Cited by in corpus (52)

• A Survey of Deep Learning-based Object Detection
• Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art
• From Handcrafted to Deep Features for Pedestrian Detection: A Survey
• Hierarchical Scene Coordinate Classification and Regression for Visual Localization
• Vision-Based Human Pose Estimation via Deep Learning: A Survey
• Deep Learning-Based Human Pose Estimation: A Survey
• SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera
• Human Body Pose Estimation for Gait Identification: A Comprehensive Survey of Datasets and Models
• The Metaverse: Survey, Trends, Novel Pipeline Ecosystem & Future Directions
• Geometric Pose Affordance: 3D Human Pose with Scene Constraints
• Single-shot 3D multi-person pose estimation in complex images
• Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
• Deep learning for 3D human pose estimation and mesh recovery: A survey
• DenseBody: Directly Regressing Dense 3D Human Pose and Shape From a Single Color Image
• Dual networks based 3D Multi-Person Pose Estimation from Monocular Video
• Universal Adversarial Perturbations: A Survey
• Single-Stage Multi-Person Pose Machines
• UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition
• PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time
• Multimodal Personality Recognition using Cross-Attention Transformer and Behaviour Encoding
• Simultaneously-Collected Multimodal Lying Pose Dataset: Towards In-Bed Human Pose Monitoring under Adverse Vision Conditions
• NIMBLE: A Non-rigid Hand Model with Bones and Muscles
• Three-Stream 3D/1D CNN for Fine-Grained Action Classification and Segmentation in Table Tennis
• Structure from Articulated Motion: Accurate and Stable Monocular 3D Reconstruction without Training Data
• VoxelPose: Towards Multi-Camera 3D Human Pose Estimation in Wild Environment
• Monocular, One-stage, Regression of Multiple 3D People
• SiCloPe: Silhouette-Based Clothed People
• HMOR: Hierarchical Multi-Person Ordinal Relations for Monocular Multi-Person 3D Pose Estimation
• From Kinematics To Dynamics: Estimating Center of Pressure and Base of Support from Video Frames of Human Motion
• Hulk: A Universal Knowledge Translator for Human-Centric Tasks
• Predicting Camera Viewpoint Improves Cross-dataset Generalization for 3D Human Pose Estimation
• Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction
• HDNet: Human Depth Estimation for Multi-Person Camera-Space Localization
• Learning Dynamics from Kinematics: Estimating 2D Foot Pressure Maps from Video Frames
• Graph and Temporal Convolutional Networks for 3D Multi-person Pose Estimation in Monocular Videos
• SMAP: Single-Shot Multi-Person Absolute 3D Pose Estimation
• DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras
• Monocular Real-Time Volumetric Performance Capture
• Single Image Human Proxemics Estimation for Visual Social Distancing
• Exploring Severe Occlusion: Multi-Person 3D Pose Estimation with Gated Convolution
• Canonical 3D Deformer Maps: Unifying parametric and non-parametric methods for dense weakly-supervised category reconstruction
• Kinematics-Guided Reinforcement Learning for Object-Aware 3D Ego-Pose Estimation
• Reconstructing 3D Human Pose by Watching Humans in the Mirror
• Selective Spatio-Temporal Aggregation Based Pose Refinement System: Towards Understanding Human Activities in Real-World Videos
• Single-view 3D Body and Cloth Reconstruction under Complex Poses
• Monocular 3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks
• PandaNet : Anchor-Based Single-Shot Multi-Person 3D Pose Estimation
• An Efficient Model-Agnostic Approach for Uncertainty Estimation in Data-Restricted Pedometric Applications
• Iterative Greedy Matching for 3D Human Pose Tracking from Multiple Views
• Learning to Train with Synthetic Humans
• CGAP2: Context and gap aware predictive pose framework for early detection of gestures
• VQ-HPS: Human Pose and Shape Estimation in a Vector-Quantized Latent Space