FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration
arXiv:2008.08324
Abstract
Although the essential nuance of human motion is often conveyed as a combination of body movements and hand gestures, the existing monocular motion capture approaches mostly focus on either body motion capture only ignoring hand parts or hand motion capture only without considering body motion. In this paper, we present FrankMocap, a motion capture system that can estimate both 3D hand and body motion from in-the-wild monocular inputs with faster speed (9.5 fps) and better accuracy than previous work. Our method works in near real-time (9.5 fps) and produces 3D body and hand motion capture outputs as a unified parametric model structure. Our method aims to capture 3D body and hand motion simultaneously from challenging in-the-wild monocular videos. To construct FrankMocap, we build the state-of-the-art monocular 3D "hand" motion capture method by taking the hand part of the whole body parametric model (SMPL-X). Our 3D hand motion capture output can be efficiently integrated to monocular body motion capture output, producing whole body motion results in a unified parrametric model structure. We demonstrate the state-of-the-art performance of our hand motion capture system in public benchmarks, and show the high quality of our whole body motion capture result in various challenging real-world scenes, including a live demo scenario.
arXiv commentDemo, Code and Models are available at https://penincillin.github.io/frank_mocap
References in corpus (6)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
- Embodied Hands: Modeling and Capturing Hands and Bodies Together
- Capturing Hands in Action using Discriminative Salient Points and Physics Simulation
- Fast and Robust Hand Tracking Using Detection-Guided Optimization
- 3D Hand Pose Tracking and Estimation Using Stereo Matching
Cited by in corpus (6)
- Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras
- Single-view 3D Body and Cloth Reconstruction under Complex Poses
- Open-World Entity Segmentation
- Self-Supervised Motion Retargeting with Safety Guarantee
- 4D Attention: Comprehensive Framework for Spatio-Temporal Gaze Mapping
- Single RGB-D Camera Teleoperation for General Robotic Manipulation