Can Action be Imitated? Learn to Reconstruct and Transfer Human Dynamics from Videos

Abstract

Given a video demonstration, can we imitate the action contained in this video? In this paper, we introduce a novel task, dubbed mesh-based action imitation. The goal of this task is to enable an arbitrary target human mesh to perform the same action shown on the video demonstration. To achieve this, a novel Mesh-based Video Action Imitation (M-VAI) method is proposed by us. M-VAI first learns to reconstruct the meshes from the given source image frames, then the initial recovered mesh sequence is fed into mesh2mesh, a mesh sequence smooth module proposed by us, to improve the temporal consistency. Finally, we imitate the actions by transferring the pose from the constructed human body to our target identity mesh. High-quality and detailed human body meshes can be generated by using our M-VAI. Extensive experiments demonstrate the feasibility of our task and the effectiveness of our proposed method.

arXiv commentaccpected by ICMR2021

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Semi-Supervised Classification with Graph Convolutional Networks
• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• Towards Accurate Markerless Human Shape and Pose Estimation over Time
• Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop