Robots That Can See: Leveraging Human Pose for Trajectory Prediction

Abstract

Anticipating the motion of all humans in dynamic environments such as homes and offices is critical to enable safe and effective robot navigation. Such spaces remain challenging as humans do not follow strict rules of motion and there are often multiple occluded entry points such as corners and doors that create opportunities for sudden encounters. In this work, we present a Transformer based architecture to predict human future trajectories in human-centric environments from input features including human positions, head orientations, and 3D skeletal keypoints from onboard in-the-wild sensory information. The resulting model captures the inherent uncertainty for future human trajectory prediction and achieves state-of-the-art performance on common prediction benchmarks and a human tracking dataset captured from a mobile robot adapted for the prediction task. Furthermore, we identify new agents with limited historical data as a major contributor to error and demonstrate the complementary nature of 3D skeletal poses in reducing prediction error in such challenging scenarios.

arXiv commentProject page: https://human-scene-transformer.github.io/

References in corpus (9)

• One Thousand and One Hours: Self-driving Motion Prediction Dataset
• JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments
• Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting
• From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting
• BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimation
• SoMoFormer: Multi-Person Pose Forecasting with Transformers
• HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving
• Wayformer: Motion Forecasting via Simple & Efficient Attention Networks
• STPOTR: Simultaneous Human Trajectory and Pose Prediction Using a Non-Autoregressive Transformer for Robot Following Ahead

Cited by in corpus (4)

• Long-Term Human Trajectory Prediction using 3D Dynamic Scene Graphs
• The Magni Human Motion Dataset: Accurate, Complex, Multi-Modal, Natural, Semantically-Rich and Contextualized
• Long-Term Prediction of Local and Global Human Motion with Occlusion Recovery
• Adaptive Interaction Network for Human Motion Prediction During Human-Robot Collaboration