Track to Reconstruct and Reconstruct to Track

Abstract

Object tracking and 3D reconstruction are often performed together, with tracking used as input for reconstruction. However, the obtained reconstructions also provide useful information for improving tracking. We propose a novel method that closes this loop, first tracking to reconstruct, and then reconstructing to track. Our approach, MOTSFusion (Multi-Object Tracking, Segmentation and dynamic object Fusion), exploits the 3D motion extracted from dynamic object reconstructions to track objects through long periods of complete occlusion and to recover missing detections. Our approach first builds up short tracklets using 2D optical flow, and then fuses these into dynamic 3D object reconstructions. The precise 3D object motion of these reconstructions is used to merge tracklets through occlusion into long-term tracks, and to locate objects when detections are missing. On KITTI, our reconstruction-based tracking reduces the number of ID switches of the initial tracklets by more than 50%, and outperforms all previous approaches for both bounding box and segmentation tracking.

arXiv commentRA-L 2020 and ICRA 2020

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras
• MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
• Robust Dense Mapping for Large-Scale Dynamic Environments
• Multi-Class Multi-Object Tracking using Changing Point Detection
• Learning Optimal Parameters for Multi-target Tracking with Contextual Interactions
• Accurate Single Stage Detector Using Recurrent Rolling Convolution
• Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud
• A metric for sets of trajectories that is practical and mathematically consistent

Cited by in corpus (24)

• HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking
• Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review
• DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association
• Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration
• TripletTrack: 3D Object Tracking using Triplet Embeddings and LSTM
• TrackMPNN: A Message Passing Graph Neural Architecture for Multi-Object Tracking
• PNAS-MOT: Multi-Modal Object Tracking with Pareto Neural Architecture Search
• Weakly-Supervised Domain Adaptation of Deep Regression Trackers via Reinforced Knowledge Distillation
• Tracking Objects as Points
• Integration of the 3D Environment for UAV Onboard Visual Object Tracking
• FlowMOT: 3D Multi-Object Tracking by Scene Flow Association
• ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation
• PointTrack++ for Effective Online Multi-Object Tracking and Segmentation
• ClusterVO: Clustering Moving Instances and Estimating Visual Odometry for Self and Surroundings
• MOTS R-CNN: Cosine-margin-triplet loss for multi-object tracking
• EagerMOT: 3D Multi-Object Tracking via Sensor Fusion
• DymSLAM:4D Dynamic Scene Reconstruction Based on Geometrical Motion Segmentation
• Segment as Points for Efficient Online Multi-Object Tracking and Segmentation
• 3D-FCT: Simultaneous 3D Object Detection and Tracking Using Feature Correlation
• MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videos
• Relation3DMOT: Exploiting Deep Affinity for 3D Multi-Object Tracking from View Aggregation
• PolyTrack: Tracking with Bounding Polygons
• Opening up Open-World Tracking
• MeNToS: Tracklets Association with a Space-Time Memory Network