ELMO: Enhanced Real-time LiDAR Motion Capture through Upsampling

Abstract

This paper introduces ELMO, a real-time upsampling motion capture framework designed for a single LiDAR sensor. Modeled as a conditional autoregressive transformer-based upsampling motion generator, ELMO achieves 60 fps motion capture from a 20 fps LiDAR point cloud sequence. The key feature of ELMO is the coupling of the self-attention mechanism with thoughtfully designed embedding modules for motion and point clouds, significantly elevating the motion quality. To facilitate accurate motion capture, we develop a one-time skeleton calibration model capable of predicting user skeleton offsets from a single-frame point cloud. Additionally, we introduce a novel data augmentation technique utilizing a LiDAR simulator, which enhances global root tracking to improve environmental understanding. To demonstrate the effectiveness of our method, we compare ELMO with state-of-the-art methods in both image-based and point cloud-based motion capture. We further conduct an ablation study to validate our design principles. ELMO's fast inference time makes it well-suited for real-time applications, exemplified in our demo video featuring live streaming and interactive gaming scenarios. Furthermore, we contribute a high-quality LiDAR-mocap synchronized dataset comprising 20 different subjects performing a range of motions, which can serve as a valuable resource for future research. The dataset and evaluation code are available at {\blue \url{https://movin3d.github.io/ELMO_SIGASIA2024/}}

published at ACM Transactions on Graphics (Proc. SIGGRAPH ASIA), 2024

References in corpus (11)

• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Character Controllers Using Motion VAEs
• MoGlow: Probabilistic and controllable motion synthesis using normalising flows
• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars
• GANimator: Neural Motion Synthesis from a Single Sequence
• Learning to Generate Diverse Dance Motions with Transformer
• LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors
• SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data
• C$\cdot$ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters
• Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control
• MOCHA: Real-Time Motion Characterization via Context Matching