Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image

Abstract

We propose a unified formulation for the problem of 3D human pose estimation from a single raw RGB image that reasons jointly about 2D joint estimation and 3D pose reconstruction to improve both tasks. We take an integrated approach that fuses probabilistic knowledge of 3D human pose with a multi-stage CNN architecture and uses the knowledge of plausible 3D landmark locations to refine the search for better 2D locations. The entire process is trained end-to-end, is extremely efficient and obtains state- of-the-art results on Human3.6M outperforming previous approaches both on 2D and 3D errors.

arXiv commentPaper presented at CVPR 17

References in corpus (12)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• Learning Human Pose Estimation Features with Convolutional Networks
• Sparse Representation for 3D Shape Estimation: A Convex Relaxation Approach
• Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video
• Predicting People's 3D Poses from Short Sequences
• Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation
• Deep Kinematic Pose Regression
• Representing Data by a Mixture of Activated Simplices

Cited by in corpus (96)

• Deep learning tools for the measurement of animal behavior in neuroscience
• Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods
• LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
• Exploiting temporal information for 3D pose estimation
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
• Self-supervised Learning of Motion Capture
• MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency
• MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation
• SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera
• Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach
• BodyNet: Volumetric Inference of 3D Human Body Shapes
• OriNet: A Fully Convolutional Network for 3D Human Pose Estimation
• Human Body Pose Estimation for Gait Identification: A Comprehensive Survey of Datasets and Models
• Deep Part Induction from Articulated Object Pairs
• EmotioNet Challenge: Recognition of facial expressions of emotion in the wild
• TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation
• Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views
• 3D Human Pose Estimation in the Wild by Adversarial Learning
• DeepMoCap: Deep Optical Motion Capture Using Multiple Depth Sensors and Retro-Reflectors
• DenseBody: Directly Regressing Dense 3D Human Pose and Shape From a Single Color Image
• MoSculp: Interactive Visualization of Shape and Time
• 3D Human Pose Estimation with Relational Networks
• Video Based Reconstruction of 3D People Models
• Optimal and Robust Category-level Perception: Object Pose and Shape Estimation from 2D and 3D Semantic Keypoints
• Learning to Estimate 3D Human Pose and Shape from a Single Color Image
• PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time
• Metric-Scale Truncation-Robust Heatmaps for 3D Human Pose Estimation
• DeepHuman: 3D Human Reconstruction from a Single Image
• SMPLR: Deep SMPL reverse for 3D human pose and shape recovery
• Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera
• A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
• Unsupervised 3D Pose Estimation with Geometric Self-Supervision
• Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
• Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision
• Integral Human Pose Regression
• RGB-based 3D Hand Pose Estimation via Privileged Learning with Depth Images
• Not All Parts Are Created Equal: 3D Pose Estimation by Modelling Bi-directional Dependencies of Body Parts
• Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosis
• Shape-Aware Human Pose and Shape Reconstruction Using Multi-View Images
• A Review on Human Pose Estimation
• Learning to Reconstruct People in Clothing from a Single RGB Camera
• FBI-Pose: Towards Bridging the Gap between 2D Images and 3D Human Poses using Forward-or-Backward Information
• Weakly-Supervised 3D Pose Estimation from a Single Image using Multi-View Consistency
• Cross View Fusion for 3D Human Pose Estimation
• Motion Guided 3D Pose Estimation from Videos
• What Face and Body Shapes Can Tell About Height
• Creating a Robot Coach for Mindfulness and Wellbeing: A Longitudinal Study
• LiveCap: Real-time Human Performance Capture from Monocular Video
• Feature Boosting Network For 3D Pose Estimation
• HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation
• Learning 3D Human Pose from Structure and Motion
• Generalizing Monocular 3D Human Pose Estimation in the Wild
• DeepSkeleton: Skeleton Map for 3D Human Pose Regression
• PISEP^2: Pseudo Image Sequence Evolution based 3D Pose Prediction
• Learning Dynamics from Kinematics: Estimating 2D Foot Pressure Maps from Video Frames
• Towards Robust Direction Invariance in Character Animation
• Single Image Human Proxemics Estimation for Visual Social Distancing
• Silhouette-Net: 3D Hand Pose Estimation from Silhouettes
• Ordinal Depth Supervision for 3D Human Pose Estimation
• Dense 3D Regression for Hand Pose Estimation
• Everybody Is Unique: Towards Unbiased Human Mesh Recovery
• Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture
• Patch-based 3D Human Pose Refinement
• Toward Marker-free 3D Pose Estimation in Lifting: A Deep Multi-view Solution
• Novel View Synthesis of Humans using Differentiable Rendering
• AIR-Act2Act: Human-human interaction dataset for teaching non-verbal social behaviors to robots
• In Perfect Shape: Certifiably Optimal 3D Shape Reconstruction from 2D Landmarks
• Motion Capture from Internet Videos
• Distill Knowledge from NRSfM for Weakly Supervised 3D Pose Learning
• 3D Human Pose Machines with Self-supervised Learning
• Deep Structure for end-to-end inverse rendering
• Heuristic Weakly Supervised 3D Human Pose Estimation
• CamLoc: Pedestrian Location Detection from Pose Estimation on Resource-constrained Smart-cameras
• 3D Human Shape Reconstruction from a Polarization Image
• Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB
• PCLs: Geometry-aware Neural Reconstruction of 3D Pose with Perspective Crop Layers
• Deep Autoencoder for Combined Human Pose Estimation and body Model Upscaling
• Coherent Reconstruction of Multiple Humans from a Single Image
• PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections
• Towards More Realistic Human-Robot Conversation: A Seq2Seq-based Body Gesture Interaction System
• Skeleton Transformer Networks: 3D Human Pose and Skinned Mesh from Single RGB Image
• MEBOW: Monocular Estimation of Body Orientation In the Wild
• Adversarial Refinement Network for Human Motion Prediction
• Discovery and recognition of motion primitives in human activities
• Semantic Estimation of 3D Body Shape and Pose using Minimal Cameras
• Explicit Spatiotemporal Joint Relation Learning for Tracking Human Pose
• A Robust Billboard-based Free-viewpoint Video Synthesizing Algorithm for Sports Scenes
• Object Properties Inferring from and Transfer for Human Interaction Motions
• ActiveMoCap: Optimized Viewpoint Selection for Active Human Motion Capture
• Deep NRSfM++: Towards Unsupervised 2D-3D Lifting in the Wild
• PoseKernelLifter: Metric Lifting of 3D Human Pose using Sound
• Iterative Greedy Matching for 3D Human Pose Tracking from Multiple Views
• Human 3D keypoints via spatial uncertainty modeling
• Joint Representation of Multiple Geometric Priors via a Shape Decomposition Model for Single Monocular 3D Pose Estimation
• TriPose: A Weakly-Supervised 3D Human Pose Estimation via Triangulation from Video