Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image

Abstract

We propose a unified formulation for the problem of 3D human pose estimation from a single raw RGB image that reasons jointly about 2D joint estimation and 3D pose reconstruction to improve both tasks. We take an integrated approach that fuses probabilistic knowledge of 3D human pose with a multi-stage CNN architecture and uses the knowledge of plausible 3D landmark locations to refine the search for better 2D locations. The entire process is trained end-to-end, is extremely efficient and obtains state- of-the-art results on Human3.6M outperforming previous approaches both on 2D and 3D errors.

Paper presented at CVPR 17

References in corpus (6)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• Deep Kinematic Pose Regression
• Representing Data by a Mixture of Activated Simplices

Cited by in corpus (95)

• Deep learning tools for the measurement of animal behavior in neuroscience
• Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods
• LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
• Exploiting temporal information for 3D pose estimation
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
• Self-supervised Learning of Motion Capture
• MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency
• MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation
• SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera
• Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach
• BodyNet: Volumetric Inference of 3D Human Body Shapes
• OriNet: A Fully Convolutional Network for 3D Human Pose Estimation
• Human Body Pose Estimation for Gait Identification: A Comprehensive Survey of Datasets and Models
• Deep Part Induction from Articulated Object Pairs
• EmotioNet Challenge: Recognition of facial expressions of emotion in the wild
• TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation
• Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views
• 3D Human Pose Estimation in the Wild by Adversarial Learning
• DeepMoCap: Deep Optical Motion Capture Using Multiple Depth Sensors and Retro-Reflectors
• DenseBody: Directly Regressing Dense 3D Human Pose and Shape From a Single Color Image
• MoSculp: Interactive Visualization of Shape and Time
• 3D Human Pose Estimation with Relational Networks
• Video Based Reconstruction of 3D People Models
• Optimal and Robust Category-level Perception: Object Pose and Shape Estimation from 2D and 3D Semantic Keypoints
• PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time
• Learning to Estimate 3D Human Pose and Shape from a Single Color Image
• DeepHuman: 3D Human Reconstruction from a Single Image
• Metric-Scale Truncation-Robust Heatmaps for 3D Human Pose Estimation
• SMPLR: Deep SMPL reverse for 3D human pose and shape recovery
• Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera
• A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
• Unsupervised 3D Pose Estimation with Geometric Self-Supervision
• Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
• Integral Human Pose Regression
• RGB-based 3D Hand Pose Estimation via Privileged Learning with Depth Images
• Not All Parts Are Created Equal: 3D Pose Estimation by Modelling Bi-directional Dependencies of Body Parts
• Shape-Aware Human Pose and Shape Reconstruction Using Multi-View Images
• A Review on Human Pose Estimation
• Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosis
• Learning to Reconstruct People in Clothing from a Single RGB Camera
• FBI-Pose: Towards Bridging the Gap between 2D Images and 3D Human Poses using Forward-or-Backward Information
• Weakly-Supervised 3D Pose Estimation from a Single Image using Multi-View Consistency
• Cross View Fusion for 3D Human Pose Estimation
• Motion Guided 3D Pose Estimation from Videos
• What Face and Body Shapes Can Tell About Height
• LiveCap: Real-time Human Performance Capture from Monocular Video
• Creating a Robot Coach for Mindfulness and Wellbeing: A Longitudinal Study
• Feature Boosting Network For 3D Pose Estimation
• HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation
• Learning 3D Human Pose from Structure and Motion
• Generalizing Monocular 3D Human Pose Estimation in the Wild
• DeepSkeleton: Skeleton Map for 3D Human Pose Regression
• Towards Robust Direction Invariance in Character Animation
• Learning Dynamics from Kinematics: Estimating 2D Foot Pressure Maps from Video Frames
• PISEP^2: Pseudo Image Sequence Evolution based 3D Pose Prediction
• Single Image Human Proxemics Estimation for Visual Social Distancing
• Silhouette-Net: 3D Hand Pose Estimation from Silhouettes
• Everybody Is Unique: Towards Unbiased Human Mesh Recovery
• Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture
• Patch-based 3D Human Pose Refinement
• Dense 3D Regression for Hand Pose Estimation
• Ordinal Depth Supervision for 3D Human Pose Estimation
• Deep Structure for end-to-end inverse rendering
• Novel View Synthesis of Humans using Differentiable Rendering
• Motion Capture from Internet Videos
• AIR-Act2Act: Human-human interaction dataset for teaching non-verbal social behaviors to robots
• In Perfect Shape: Certifiably Optimal 3D Shape Reconstruction from 2D Landmarks
• Distill Knowledge from NRSfM for Weakly Supervised 3D Pose Learning
• 3D Human Pose Machines with Self-supervised Learning
• Toward Marker-free 3D Pose Estimation in Lifting: A Deep Multi-view Solution
• Skeleton Transformer Networks: 3D Human Pose and Skinned Mesh from Single RGB Image
• CamLoc: Pedestrian Location Detection from Pose Estimation on Resource-constrained Smart-cameras
• Coherent Reconstruction of Multiple Humans from a Single Image
• Deep Autoencoder for Combined Human Pose Estimation and body Model Upscaling
• Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB
• Heuristic Weakly Supervised 3D Human Pose Estimation
• PCLs: Geometry-aware Neural Reconstruction of 3D Pose with Perspective Crop Layers
• PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections
• 3D Human Shape Reconstruction from a Polarization Image
• Towards More Realistic Human-Robot Conversation: A Seq2Seq-based Body Gesture Interaction System
• MEBOW: Monocular Estimation of Body Orientation In the Wild
• PoseKernelLifter: Metric Lifting of 3D Human Pose using Sound
• Iterative Greedy Matching for 3D Human Pose Tracking from Multiple Views
• Discovery and recognition of motion primitives in human activities
• TriPose: A Weakly-Supervised 3D Human Pose Estimation via Triangulation from Video
• Semantic Estimation of 3D Body Shape and Pose using Minimal Cameras
• Human 3D keypoints via spatial uncertainty modeling
• A Robust Billboard-based Free-viewpoint Video Synthesizing Algorithm for Sports Scenes
• ActiveMoCap: Optimized Viewpoint Selection for Active Human Motion Capture
• Deep NRSfM++: Towards Unsupervised 2D-3D Lifting in the Wild
• Object Properties Inferring from and Transfer for Human Interaction Motions
• Explicit Spatiotemporal Joint Relation Learning for Tracking Human Pose
• Joint Representation of Multiple Geometric Priors via a Shape Decomposition Model for Single Monocular 3D Pose Estimation
• Adversarial Refinement Network for Human Motion Prediction