Self-Attentive 3D Human Pose and Shape Estimation from Videos

Abstract

We consider the task of estimating 3D human pose and shape from videos. While existing frame-based approaches have made significant progress, these methods are independently applied to each image, thereby often leading to inconsistent predictions. In this work, we present a video-based learning algorithm for 3D human pose and shape estimation. The key insights of our method are two-fold. First, to address the inconsistent temporal prediction issue, we exploit temporal information in videos and propose a self-attention module that jointly considers short-range and long-range dependencies across frames, resulting in temporally coherent estimations. Second, we model human motion with a forecasting module that allows the transition between adjacent frames to be smooth. We evaluate our method on the 3DPW, MPI-INF-3DHP, and Human3.6M datasets. Extensive experimental results show that our algorithm performs favorably against the state-of-the-art methods.

arXiv commentThis paper is under consideration at Computer Vision and Image Understanding

References in corpus (8)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• On the difficulty of training Recurrent Neural Networks
• The Kinetics Human Action Video Dataset
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Sim2real transfer learning for 3D human pose estimation: motion to the rescue
• Synthetic Training for Accurate 3D Human Pose and Shape Estimation in the Wild
• Unsupervised 3D Pose Estimation with Geometric Self-Supervision
• Show, Match and Segment: Joint Weakly Supervised Learning of Semantic Matching and Object Co-segmentation