DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency

Abstract

We present an unsupervised learning framework for simultaneously training single-view depth prediction and optical flow estimation models using unlabeled video sequences. Existing unsupervised methods often exploit brightness constancy and spatial smoothness priors to train depth or flow models. In this paper, we propose to leverage geometric consistency as additional supervisory signals. Our core idea is that for rigid regions we can use the predicted scene depth and camera motion to synthesize 2D optical flow by backprojecting the induced 3D scene flow. The discrepancy between the rigid flow (from depth prediction and camera motion) and the estimated flow (from optical flow model) allows us to impose a cross-task consistency loss. While all the networks are jointly optimized during training, they can be applied independently at test time. Extensive experiments demonstrate that our depth and flow models compare favorably with state-of-the-art unsupervised methods.

arXiv commentECCV 2018. Project website: http://yuliang.vision/DF-Net/ Code: https://github.com/vt-vl-lab/DF-Net

References in corpus (18)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Deep Residual Learning for Image Recognition
• An Overview of Multi-Task Learning in Deep Neural Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• SfM-Net: Learning of Structure and Motion from Video
• Virtual Worlds as Proxy for Multi-Object Tracking Analysis
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• Unsupervised Learning of Geometry with Edge-aware Depth-Normal Consistency
• Single-Image Depth Perception in the Wild
• Semi-Supervised Deep Learning for Monocular Depth Map Prediction
• MegaDepth: Learning Single-View Depth Prediction from Internet Photos
• Monocular Object Instance Segmentation and Depth Ordering with CNNs
• DeepMVS: Learning Multi-view Stereopsis
• Learning Single-Image Depth from Videos using Quality Assessment Networks