TW-SMNet: Deep Multitask Learning of Tele-Wide Stereo Matching

Abstract

In this paper, we introduce the problem of estimating the real world depth of elements in a scene captured by two cameras with different field of views, where the first field of view (FOV) is a Wide FOV (WFOV) captured by a wide angle lens, and the second FOV is contained in the first FOV and is captured by a tele zoom lens. We refer to the problem of estimating the inverse depth for the union of FOVs, while leveraging the stereo information in the overlapping FOV, as Tele-Wide Stereo Matching (TW-SM). We propose different deep learning solutions to the TW-SM problem. Since the disparity is proportional to the inverse depth, we train stereo matching disparity estimation (SMDE) networks to estimate the disparity for the union WFOV. We further propose an end-to-end deep multitask tele-wide stereo matching neural network (MT-TW-SMNet), which simultaneously learns the SMDE task for the overlapped Tele FOV and the single image inverse depth estimation (SIDE) task for the WFOV. Moreover, we design multiple methods for the fusion of the SMDE and SIDE networks. We evaluate the performance of TW-SM on the popular KITTI and SceneFlow stereo datasets, and demonstrate its practicality by synthesizing the Bokeh effect on the WFOV from a tele-wide stereo image pair.

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Fully Convolutional Networks for Semantic Segmentation
• A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• ParseNet: Looking Wider to See Better
• Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
• DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
• Stacked Deconvolutional Network for Semantic Segmentation
• Learning to Refine Object Segments
• Label Refinement Network for Coarse-to-Fine Semantic Segmentation
• AMNet: Deep Atrous Multiscale Stereo Disparity Estimation Networks
• Combining Stereo Disparity and Optical Flow for Basic Scene Flow
• Single image depth estimation by dilated deep residual convolutional neural network and soft-weight-sum inference
• Improved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning

Cited by in corpus (1)

• Robust Full-FoV Depth Estimation in Tele-wide Camera System