View-Invariant, Occlusion-Robust Probabilistic Embedding for Human Pose

Abstract

Recognition of human poses and actions is crucial for autonomous systems to interact smoothly with people. However, cameras generally capture human poses in 2D as images and videos, which can have significant appearance variations across viewpoints that make the recognition tasks challenging. To address this, we explore recognizing similarity in 3D human body poses from 2D information, which has not been well-studied in existing works. Here, we propose an approach to learning a compact view-invariant embedding space from 2D body joint keypoints, without explicitly predicting 3D poses. Input ambiguities of 2D poses from projection and occlusion are difficult to represent through a deterministic mapping, and therefore we adopt a probabilistic formulation for our embedding space. Experimental results show that our embedding model achieves higher accuracy when retrieving similar poses across different camera views, in comparison with 3D pose estimation models. We also show that by training a simple temporal embedding model, we achieve superior performance on pose sequence retrieval and largely reduce the embedding dimension from stacking frame-based embeddings for efficient large-scale retrieval. Furthermore, in order to enable our embeddings to work with partially visible input, we further investigate different keypoint occlusion augmentation strategies during training. We demonstrate that these occlusion augmentations significantly improve retrieval performance on partial 2D input poses. Results on action recognition and video alignment demonstrate that using our embeddings without any additional training achieves competitive performance relative to other models specifically trained for each task.

arXiv commentAccepted to International Journal of Computer Vision (IJCV). Code is available at https://github.com/google-research/google-research/tree/master/poem. Video synchronization results are available at https://drive.google.com/corp/drive/folders/1nhPuEcX4Lhe6iK3nv84cvSCov2eJ52Xy. arXiv admin note: text overlap with arXiv:1912.01001

References in corpus (18)

• Auto-Encoding Variational Bayes
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Semi-Supervised Classification with Graph Convolutional Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• In Defense of the Triplet Loss for Person Re-Identification
• Semantic Graph Convolutional Networks for 3D Human Pose Regression
• Learning Descriptors for Object Recognition and 3D Pose Estimation
• Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking
• Cascaded deep monocular 3D human pose estimation with evolutionary training data
• Word Representations via Gaussian Embedding
• Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
• Unsupervised Learning of View-invariant Action Representations
• Modeling Uncertainty with Hedged Instance Embedding
• Triplet-Center Loss for Multi-View 3D Object Retrieval
• Synthetic Occlusion Augmentation with Volumetric Heatmaps for the 2018 ECCV PoseTrack Challenge on 3D Human Pose Estimation
• Pose Embeddings: A Deep Architecture for Learning to Match Human Poses

Cited by in corpus (1)

• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition