Neuromorphic High-Frequency 3D Dancing Pose Estimation in Dynamic Environment

Abstract

As a beloved sport worldwide, dancing is getting integrated into traditional and virtual reality-based gaming platforms nowadays. It opens up new opportunities in the technology-mediated dancing space. These platforms primarily rely on passive and continuous human pose estimation as an input capture mechanism. Existing solutions are mainly based on RGB or RGB-Depth cameras for dance games. The former suffers in low-lighting conditions due to the motion blur and low sensitivity, while the latter is too power-hungry, has a low frame rate, and has limited working distance. With ultra-low latency, energy efficiency, and wide dynamic range characteristics, the event camera is a promising solution to overcome these shortcomings. We propose YeLan, an event camera-based 3-dimensional high-frequency human pose estimation(HPE) system that survives low-lighting conditions and dynamic backgrounds. We collected the world's first event camera dance dataset and developed a fully customizable motion-to-event physics-aware simulator. YeLan outperforms the baseline models in these challenging conditions and demonstrated robustness against different types of clothing, background motion, viewing angle, occlusion, and lighting fluctuations.

References in corpus (7)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Event-based Vision: A Survey
• Robust Real-World Image Super-Resolution against Adversarial Attacks
• The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
• Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods
• ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation
• Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation