A Survey on Deep Learning Techniques for Stereo-based Depth Estimation

Abstract

Estimating depth from RGB images is a long-standing ill-posed problem, which has been explored for decades by the computer vision, graphics, and machine learning communities. Among the existing techniques, stereo matching remains one of the most widely used in the literature due to its strong connection to the human binocular system. Traditionally, stereo-based depth estimation has been addressed through matching hand-crafted features across multiple images. Despite the extensive amount of research, these traditional techniques still suffer in the presence of highly textured areas, large uniform regions, and occlusions. Motivated by their growing success in solving various 2D and 3D vision problems, deep learning for stereo-based depth estimation has attracted growing interest from the community, with more than 150 papers published in this area between 2014 and 2019. This new generation of methods has demonstrated a significant leap in performance, enabling applications such as autonomous driving and augmented reality. In this article, we provide a comprehensive survey of this new and continuously growing field of research, summarize the most commonly used pipelines, and discuss their benefits and limitations. In retrospect of what has been achieved so far, we also conjecture what the future may hold for deep learning-based stereo for depth estimation research.

References in corpus (28)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Conditional Random Fields as Recurrent Neural Networks
• A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• DARTS: Differentiable Architecture Search
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
• Computing the Stereo Matching Cost with a Convolutional Neural Network
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• Deep Convolutional Inverse Graphics Network
• The ApolloScape Open Dataset for Autonomous Driving and its Application
• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis
• SfM-Net: Learning of Structure and Motion from Video
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• A2D2: Audi Autonomous Driving Dataset
• Descriptor Matching with Convolutional Neural Networks: a Comparison to SIFT
• Learning a Multi-View Stereo Machine
• What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
• Learning Local Image Descriptors with Deep Siamese and Triplet Convolutional Networks by Minimising Global Loss Functions
• PN-Net: Conjoined Triple Deep Network for Learning Local Image Descriptors
• Self-Supervised Learning for Stereo Matching with Self-Improving Ability
• Look Wider to Match Image Patches with Convolutional Neural Networks
• Practical Deep Stereo (PDS): Toward applications-friendly deep stereo matching
• Vortex Pooling: Improving Context Representation in Semantic Segmentation
• Deep Stereo Matching with Explicit Cost Aggregation Sub-Architecture

Cited by in corpus (17)

• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey
• Neural Markov Random Field for Stereo Matching
• Vision-based Learning for Drones: A Survey
• UAMD-Net: A Unified Adaptive Multimodal Neural Network for Dense Depth Completion
• IRS: A Large Naturalistic Indoor Robotics Stereo Dataset to Train Deep Models for Disparity and Surface Normal Estimation
• DoubleStar: Long-Range Attack Towards Depth Estimation based Obstacle Avoidance in Autonomous Systems
• An evaluation of Deep Learning based stereo dense matching dataset shift from aerial images and a large scale stereo dataset
• Regularising disparity estimation via multi task learning with structured light reconstruction
• Real-time FPGA implementation of the Semi-Global Matching stereo vision algorithm for a 4K/UHD video stream
• Normal-guided Detail-Preserving Neural Implicit Function for High-Fidelity 3D Surface Reconstruction
• 360° Stereo Image Composition with Depth Adaption
• Monocular Spherical Depth Estimation with Explicitly Connected Weak Layout Cues
• Icy Moon Surface Simulation and Stereo Depth Estimation for Sampling Autonomy
• The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation
• Convolutional Neural Shading for High-Quality 3D Reconstruction from Multi-View Images
• Learnable Triangulation for Deep Learning-based 3D Reconstruction of Objects of Arbitrary Topology from Single RGB Images