OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
arXiv:1812.08008
Abstract
Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime approach to detect the 2D pose of multiple people in an image. The proposed method uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. This bottom-up system achieves high accuracy and realtime performance, regardless of the number of people in the image. In previous work, PAFs and body part location estimation were refined simultaneously across training stages. We demonstrate that a PAF-only refinement rather than both PAF and body part location refinement results in a substantial increase in both runtime performance and accuracy. We also present the first combined body and foot keypoint detector, based on an internal annotated foot dataset that we have publicly released. We show that the combined detector not only reduces the inference time compared to running them sequentially, but also maintains the accuracy of each component individually. This work has culminated in the release of OpenPose, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints.
Journal version of arXiv:1611.08050, with better accuracy and faster speed, release a new foot keypoint dataset: https://cmu-perceptual-computing-lab.github.io/foot_keypoint_dataset/
References in corpus (7)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- DeepPose: Human Pose Estimation via Deep Neural Networks
- VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
- Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
- Associative Embedding: End-to-End Learning for Joint Detection and Grouping
- Human pose estimation via Convolutional Part Heatmap Regression
- Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations
Cited by in corpus (88)
- A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
- Unpaired Motion Style Transfer from Video to Animation
- JA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention
- Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
- MoVi: A Large Multipurpose Motion and Video Dataset
- Flood severity mapping from Volunteered Geographic Information by interpreting water level from images containing people: a case study of Hurricane Harvey
- Modeling Engagement in Long-Term, In-Home Socially Assistive Robot Interventions for Children with Autism Spectrum Disorders
- Pedestrian Models for Autonomous Driving Part I: Low-Level Models, from Sensing to Tracking
- A Survey on 3D Skeleton-Based Action Recognition Using Learning Method
- Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation
- A cow structural model for video analytics of cow health
- Improving Social Awareness Through DANTE: A Deep Affinity Network for Clustering Conversational Interactants
- Identity and Posture Recognition in Smart Beds with Deep Multitask Learning
- Fast Graph Fourier Transforms Based on Graph Symmetry and Bipartition
- AggPose: Deep Aggregation Vision Transformer for Infant Pose Estimation
- Real-Time Multi-View 3D Human Pose Estimation using Semantic Feedback to Smart Edge Sensors
- BosphorusSign22k Sign Language Recognition Dataset
- Dual networks based 3D Multi-Person Pose Estimation from Monocular Video
- Cross-domain Correspondence Learning for Exemplar-based Image Translation
- Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition
- Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild
- Analyzing Data Efficiency and Performance of Machine Learning Algorithms for Assessing Low Back Pain Physical Rehabilitation Exercises
- Unsupervised Motion Retargeting for Human-Robot Imitation
- Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey
- Human Motion Transfer with 3D Constraints and Detail Enhancement
- DanceIt: Music-inspired Dancing Video Synthesis
- Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
- Gesture-to-Gesture Translation in the Wild via Category-Independent Conditional Maps
- CenterFace: Joint Face Detection and Alignment Using Face as Point
- Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
- Learning to Navigate Using Mid-Level Visual Priors
- Focusing and Diffusion: Bidirectional Attentive Graph Convolutional Networks for Skeleton-based Action Recognition
- Human Mesh Recovery from Multiple Shots
- Multiview Detection with Feature Perspective Transformation
- Human Motion Transfer from Poses in the Wild
- Pose Representations for Deep Skeletal Animation
- Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction
- Foley Music: Learning to Generate Music from Videos
- Quantitative analysis of robot gesticulation behavior
- AOL: Adaptive Online Learning for Human Trajectory Prediction in Dynamic Video Scenes
- Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation
- Learning Multi-Granular Spatio-Temporal Graph Network for Skeleton-based Action Recognition
- Anyone here? Smart embedded low-resolution omnidirectional video sensor to measure room occupancy
- 4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras
- Rotation-invariant Mixed Graphical Model Network for 2D Hand Pose Estimation
- Adaptable Human Intention and Trajectory Prediction for Human-Robot Collaboration
- PT-VTON: an Image-Based Virtual Try-On Network with Progressive Pose Attention Transfer
- Vectorizing World Buildings: Planar Graph Reconstruction by Primitive Detection and Relationship Inference
- Teaching Robots Novel Objects by Pointing at Them
- Multi-person 3D pose estimation from unlabelled data
- Solving Service Robot Tasks: UT Austin Villa@Home 2019 Team Report
- Estimating Blink Probability for Highlight Detection in Figure Skating Videos
- Aggregation and Finetuning for Clothes Landmark Detection
- Whole-Body Human Pose Estimation in the Wild
- Vision based body gesture meta features for Affective Computing
- Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras
- Companion Unmanned Aerial Vehicles: A Survey
- S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling
- Flow Guided Transformable Bottleneck Networks for Motion Retargeting
- LASR: Learning Articulated Shape Reconstruction from a Monocular Video
- Unselfie: Translating Selfies to Neutral-pose Portraits in the Wild
- Key Frame Proposal Network for Efficient Pose Estimation in Videos
- PI-Net: Pose Interacting Network for Multi-Person Monocular 3D Pose Estimation
- Multi-person 3D Pose Estimation in Crowded Scenes Based on Multi-View Geometry
- Hierarchical Kinematic Human Mesh Recovery
- Nonparametric Structure Regularization Machine for 2D Hand Pose Estimation
- Exploiting Ergonomic Priors in Human-to-Robot Task Transfer
- WildGait: Learning Gait Representations from Raw Surveillance Streams
- Towards More Realistic Human-Robot Conversation: A Seq2Seq-based Body Gesture Interaction System
- LSTA-Net: Long short-term Spatio-Temporal Aggregation Network for Skeleton-based Action Recognition
- Explicit behaviors affected by driver's trust in a driving automation system
- A ROS-based Framework for Monitoring Human and Robot Conditions in a Human-Multi-robot Team
- Lets Play Music: Audio-driven Performance Video Generation
- Multi-Person Full Body Pose Estimation
- Video Affective Effects Prediction with Multi-modal Fusion and Shot-Long Temporal Context
- Multi Scale Temporal Graph Networks For Skeleton-based Action Recognition
- PoseKernelLifter: Metric Lifting of 3D Human Pose using Sound
- Self-Attentive 3D Human Pose and Shape Estimation from Videos
- CoreUI: Interactive Core Training System with 3D Human Shape
- PoseRN: A 2D pose refinement network for bias-free multi-view 3D human pose estimation
- Skeleon-Based Typing Style Learning For Person Identification
- Skeleton-based Approaches based on Machine Vision: A Survey
- Feature Sharing Cooperative Network for Semantic Segmentation
- Learning Whole-Body Human-Robot Haptic Interaction in Social Contexts
- Towards Large-Scale Data Mining for Data-Driven Analysis of Sign Languages
- SegCodeNet: Color-Coded Segmentation Masks for Activity Detection from Wearable Cameras
- Think about boundary: Fusing multi-level boundary information for landmark heatmap regression
- StereOBJ-1M: Large-scale Stereo Image Dataset for 6D Object Pose Estimation