JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments

Abstract

We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot. The dataset includes 64 minutes of annotated multimodal sensor data including stereo cylindrical 360${}^{\circ }$ RGB video at 15 fps, 3D point clouds from two Velodyne 16 Lidars, line 3D point clouds from two Sick Lidars, audio signal, RGB-D video at 30 fps, 360${}^{\circ }$ spherical image from a fisheye camera and encoder values from the robot's wheels. Our dataset incorporates data from traditionally underrepresented scenes such as indoor environments and pedestrian areas, all from the ego-perspective of the robot, both stationary and navigating. The dataset has been annotated with over 2.3 million bounding boxes spread over 5 individual cameras and 1.8 million associated 3D cuboids around all people in the scenes totaling over 3500 time consistent trajectories. Together with our dataset and the annotations, we launch a benchmark and metrics for 2D and 3D person detection and tracking. With this dataset, which we plan on extending with further types of annotation in the future, we hope to provide a new source of data and a test-bench for research in the areas of egocentric robot vision, autonomous navigation, and all perceptual tasks around social robotics in human environments.

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• YOLOv3: An Incremental Improvement
• MOT16: A Benchmark for Multi-Object Tracking
• Tracking without bells and whistles
• MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking
• The ApolloScape Open Dataset for Autonomous Driving and its Application
• MOT20: A benchmark for multi object tracking in crowded scenes
• AlignedReID: Surpassing Human-Level Performance in Person Re-Identification
• Scalability in Perception for Autonomous Driving: Waymo Open Dataset
• TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents
• CVPR19 Tracking and Detection Challenge: How crowded can it get?
• Improving Social Awareness Through DANTE: A Deep Affinity Network for Clustering Conversational Interactants

Cited by in corpus (15)

• BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
• Robots That Can See: Leveraging Human Pose for Trajectory Prediction
• Long-Term Human Trajectory Prediction using 3D Dynamic Scene Graphs
• Vision-based Multi-future Trajectory Prediction: A Survey
• HEIGHT: Heterogeneous Interaction Graph Transformer for Robot Navigation in Crowded and Constrained Environments
• ArTIST: Autoregressive Trajectory Inpainting and Scoring for Tracking
• Learning Social Navigation from Demonstrations with Conditional Neural Processes
• Autonomous Navigation in Dynamic Human Environments with an Embedded 2D LiDAR-based Person Tracker
• HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videos
• SICNav-Diffusion: Safe and Interactive Crowd Navigation with Diffusion Trajectory Predictions
• FROG: A new people detection dataset for knee-high 2D range finders
• TANet++: Triple Attention Network with Filtered Pointcloud on 3D Detection
• On the Sensory Commutativity of Action Sequences for Embodied Agents
• Self-Supervised Person Detection in 2D Range Data using a Calibrated Camera
• HHI-Assist: A Dataset and Benchmark of Human-Human Interaction in Physical Assistance Scenario