ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation
arXiv:2012.05258
Abstract
In this paper, we present ViP-DeepLab, a unified model attempting to tackle the long-standing and challenging inverse projection problem in vision, which we model as restoring the point clouds from perspective image sequences while providing each point with instance-level semantic interpretations. Solving this problem requires the vision models to predict the spatial location, semantic class, and temporally consistent instance label for each 3D point. ViP-DeepLab approaches it by jointly performing monocular depth estimation and video panoptic segmentation. We name this joint task as Depth-aware Video Panoptic Segmentation, and propose a new evaluation metric along with two derived datasets for it, which will be made available to the public. On the individual sub-tasks, ViP-DeepLab also achieves state-of-the-art results, outperforming previous methods by 5.1% VPQ on Cityscapes-VPS, ranking 1st on the KITTI monocular depth estimation benchmark, and 1st on KITTI MOTS pedestrian. The datasets and the evaluation codes are made publicly available.
Video: https://youtu.be/XR4HFiwwao0 GitHub: https://github.com/joe-siyuan-qiao/ViP-DeepLab
References in corpus (12)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Simple Online and Realtime Tracking
- Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- AutoAugment: Learning Augmentation Policies from Data
- From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
- DeeperLab: Single-Shot Image Parser
- DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
- Track to Reconstruct and Reconstruct to Track
- Learning to Fuse Things and Stuff
- Integrated Object Detection and Tracking with Tracklet-Conditioned Detection
- ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation
- Bidirectional Attention Network for Monocular Depth Estimation