Rethinking Monocular Depth Estimation with Adversarial Training

Abstract

Monocular depth estimation is an extensively studied computer vision problem with a vast variety of applications. Deep learning-based methods have demonstrated promise for both supervised and unsupervised depth estimation from monocular images. Most existing approaches treat depth estimation as a regression problem with a local pixel-wise loss function. In this work, we innovate beyond existing approaches by using adversarial training to learn a context-aware, non-local loss function. Such an approach penalizes the joint configuration of predicted depth values at the patch-level instead of the pixel-level, which allows networks to incorporate more global information. In this framework, the generator learns a mapping between RGB images and its corresponding depth map, while the discriminator learns to distinguish depth map and RGB pairs from ground truth. This conditional GAN depth estimation framework is stabilized using spectral normalization to prevent mode collapse when learning from diverse datasets. We test this approach using a diverse set of generators that include U-Net and joint CNN-CRF. We benchmark this approach on the NYUv2, Make3D and KITTI datasets, and observe that adversarial training reduces relative error by several fold, achieving state-of-the-art performance.

References in corpus (10)

• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Semantic Segmentation using Adversarial Networks
• Monocular Depth Estimation using Multi-Scale Continuous CRFs as Sequential Deep Networks
• DiscrimNet: Semi-Supervised Action Recognition from Videos using Generative Adversarial Networks

Cited by in corpus (14)

• Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration
• Deep learning in biomedical optics
• Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey
• Enforcing geometric constraints of virtual normal for depth prediction
• Rapid tissue oxygenation mapping from snapshot structured-light images with adversarial deep learning
• On the Benefit of Adversarial Training for Monocular Depth Estimation
• Attention-based Context Aggregation Network for Monocular Depth Estimation
• SLAM Endoscopy enhanced by adversarial depth prediction
• Speckle illumination SFDI for projector-free optical property mapping
• Exploiting temporal consistency for real-time video depth estimation
• Deep Learning based Monocular Depth Prediction: Datasets, Methods and Applications
• Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
• Vision: A Deep Learning Approach to provide walking assistance to the visually impaired
• Y-GAN: A Generative Adversarial Network for Depthmap Estimation from Multi-camera Stereo Images