PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing

Abstract

Depth estimation and scene parsing are two particularly important tasks in visual scene understanding. In this paper we tackle the problem of simultaneous depth estimation and scene parsing in a joint CNN. The task can be typically treated as a deep multi-task learning problem [42]. Different from previous methods directly optimizing multiple tasks given the input training data, this paper proposes a novel multi-task guided prediction-and-distillation network (PAD-Net), which first predicts a set of intermediate auxiliary tasks ranging from low level to high level, and then the predictions from these intermediate auxiliary tasks are utilized as multi-modal input via our proposed multi-modal distillation modules for the final tasks. During the joint learning, the intermediate tasks not only act as supervision for learning more robust deep representations but also provide rich multi-modal information for improving the final tasks. Extensive experiments are conducted on two challenging datasets (i.e. NYUD-v2 and Cityscapes) for both the depth estimation and scene parsing tasks, demonstrating the effectiveness of the proposed approach.

arXiv commentAccepted at CVPR 2018

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Fully Convolutional Networks for Semantic Segmentation
• Conditional Random Fields as Recurrent Neural Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Recurrent Models of Visual Attention
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Fully Convolutional Networks for Semantic Segmentation
• Learning Deconvolution Network for Semantic Segmentation
• DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Pyramid Scene Parsing Network
• Semantic Image Segmentation via Deep Parsing Network
• Attention to Scale: Scale-aware Semantic Image Segmentation
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• Learning Deep Structured Multi-Scale Features using Attention-Gated CRFs for Contour Prediction
• RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation
• Analyzing Modular CNN Architectures for Joint Depth Prediction and Semantic Segmentation
• Monocular Depth Estimation with Hierarchical Fusion of Dilated CNNs and Soft-Weighted-Sum Inference

Cited by in corpus (15)

• Polarized Self-Attention: Towards High-quality Pixel-wise Regression
• On Deep Learning Techniques to Boost Monocular Depth Estimation for Autonomous Navigation
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation
• Three-Filters-to-Normal: An Accurate and Ultrafast Surface Normal Estimator
• Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation
• Multi-Modal Multi-Task (3MT) Road Segmentation
• Distribution-aware Margin Calibration for Semantic Segmentation in Images
• Disentangled Image Matting
• Multi-task Video Enhancement for Dental Interventions
• 3D-to-2D Distillation for Indoor Scene Parsing
• Structure-Attentioned Memory Network for Monocular Depth Estimation
• TriangleNet: Edge Prior Augmented Network for Semantic Segmentation through Cross-Task Consistency
• SIGNet: Semantic Instance Aided Unsupervised 3D Geometry Perception
• Monocular Outdoor Semantic Mapping with a Multi-task Network
• Incremental Scene Synthesis