Learning a Spatio-Temporal Embedding for Video Instance Segmentation

Abstract

We present a novel embedding approach for video instance segmentation. Our method learns a spatio-temporal embedding integrating cues from appearance, motion, and geometry; a 3D causal convolutional network models motion, and a monocular self-supervised depth loss models geometry. In this embedding space, video-pixels of the same instance are clustered together while being separated from other instances, to naturally track instances over time without any complex post-processing. Our network runs in real-time as our architecture is entirely causal - we do not incorporate information from future frames, contrary to previous methods. We show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset.

References in corpus (4)

• Semantic Instance Segmentation with a Discriminative Loss Function
• Semantic Instance Segmentation via Deep Metric Learning
• Video Instance Segmentation
• Towards a Principled Integration of Multi-Camera Re-Identification and Tracking through Optimal Bayes Filters

Cited by in corpus (6)

• STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos
• PointTrack++ for Effective Online Multi-Object Tracking and Segmentation
• Are We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey and Experimental Study
• Segment as Points for Efficient Online Multi-Object Tracking and Segmentation
• Weakly Supervised Instance Segmentation for Videos with Temporal Mask Consistency
• VideoClick: Video Object Segmentation with a Single Click