End-to-End Learning of Geometry and Context for Deep Stereo Regression

Abstract

We propose a novel deep learning architecture for regressing disparity from a rectified pair of stereo images. We leverage knowledge of the problem's geometry to form a cost volume using deep feature representations. We learn to incorporate contextual information using 3-D convolutions over this volume. Disparity values are regressed from the cost volume using a proposed differentiable soft argmin operation, which allows us to train our method end-to-end to sub-pixel accuracy without any additional post-processing or regularization. We evaluate our method on the Scene Flow and KITTI datasets and on KITTI we set a new state-of-the-art benchmark, while being significantly faster than competing approaches.

References in corpus (7)

• Deep Residual Learning for Image Recognition
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• Computing the Stereo Matching Cost with a Convolutional Neural Network
• Fully Convolutional Networks for Semantic Segmentation
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Learning to Compare Image Patches via Convolutional Neural Networks

Cited by in corpus (114)

• Monocular Depth Estimation Based On Deep Learning: An Overview
• Hierarchical Neural Architecture Search for Deep Stereo Matching
• Unsupervised Learning of Depth and Ego-Motion from Video
• Learning a Multi-View Stereo Machine
• Self-Supervised Learning for Stereo Matching with Self-Improving Ability
• DeepV2D: Video to Depth with Differentiable Structure from Motion
• BA-Net: Dense Bundle Adjustment Network
• GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
• Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines
• Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction
• Spatial-Angular Attention Network for Light Field Reconstruction
• Pyramid Stereo Matching Network
• Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
• MVSNet: Depth Inference for Unstructured Multi-view Stereo
• Unsupervised Monocular Depth Learning in Dynamic Scenes
• Self-supervised Learning for Video Correspondence Flow
• Cascade Residual Learning: A Two-stage Convolutional Neural Network for Stereo Matching
• GA-Net: Guided Aggregation Net for End-to-end Stereo Matching
• Wasserstein Distances for Stereo Disparity Estimation
• Stereo R-CNN based 3D Object Detection for Autonomous Driving
• Learning for Disparity Estimation through Feature Constancy
• DeepMVS: Learning Multi-view Stereopsis
• AANet: Adaptive Aggregation Network for Efficient Stereo Matching
• DeepSignals: Predicting Intent of Drivers Through Visual Signals
• Deep Robust Single Image Depth Estimation Neural Network Using Scene Understanding
• Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera
• DSGN: Deep Stereo Geometry Network for 3D Object Detection
• Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspots
• BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks
• Deep Stereo Matching with Explicit Cost Aggregation Sub-Architecture
• Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
• Deep Learning for Multi-View Stereo via Plane Sweep: A Survey
• Hierarchical Discrete Distribution Decomposition for Match Density Estimation
• Deep Multi-View Stereo gone wild
• Superpixel Segmentation with Fully Convolutional Networks
• Disp R-CNN: Stereo 3D Object Detection via Shape Prior Guided Instance Disparity Estimation
• SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction
• AdaStereo: A Simple and Efficient Approach for Adaptive Stereo Matching
• Learning monocular depth estimation infusing traditional stereo knowledge
• CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching
• FADNet: A Fast and Accurate Network for Disparity Estimation
• Left-Right Comparative Recurrent Model for Stereo Matching
• Anytime Stereo Image Depth Estimation on Mobile Devices
• Self-Supervised Learning of Depth and Ego-motion with Differentiable Bundle Adjustment
• ES-Net: An Efficient Stereo Matching Network
• A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-view Stereo Reconstruction from An Open Aerial Dataset
• InverseRenderNet: Learning single image inverse rendering
• Parallax Attention for Unsupervised Stereo Correspondence Learning
• 360SD-Net: 360° Stereo Depth Estimation with Learnable Cost Volume
• Improving Deep Stereo Network Generalization with Geometric Priors
• Towards Practical Implementations of Person Re-Identification from Full Video Frames
• KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects
• SENSE: a Shared Encoder Network for Scene-flow Estimation
• GONet: A Semi-Supervised Deep Learning Approach For Traversability Estimation
• Learning Independent Object Motion from Unlabelled Stereoscopic Videos
• Group-wise Correlation Stereo Network
• Deep Rigid Instance Scene Flow
• YOLOStereo3D: A Step Back to 2D for Efficient Stereo 3D Detection
• SymmNet: A Symmetric Convolutional Neural Network for Occlusion Detection
• Learning Inverse Depth Regression for Multi-View Stereo with Correlation Cost Volume
• On the Importance of Stereo for Accurate Depth Estimation: An Efficient Semi-Supervised Deep Neural Network Approach
• Point-Based Multi-View Stereo Network
• Learning to Detect 3D Reflection Symmetry for Single-View Reconstruction
• Hierarchical Deep Stereo Matching on High-resolution Images
• RAFT-3D: Scene Flow using Rigid-Motion Embeddings
• L3C-Stereo: Lossless Compression for Stereo Images
• End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization
• Show, Match and Segment: Joint Weakly Supervised Learning of Semantic Matching and Object Co-segmentation
• Bilateral Grid Learning for Stereo Matching Networks
• OmniSLAM: Omnidirectional Localization and Dense Mapping for Wide-baseline Multi-camera Systems
• Zoom and Learn: Generalizing Deep Stereo Matching to Novel Domains
• Learning Dense Stereo Matching for Digital Surface Models from Satellite Imagery
• Adjusting Bias in Long Range Stereo Matching: A semantics guided approach
• Deep-Learning Assisted High-Resolution Binocular Stereo Depth Reconstruction
• Flow2Stereo: Effective Self-Supervised Learning of Optical Flow and Stereo Matching
• Normal Assisted Stereo Depth Estimation
• Joint Learning of Semantic Alignment and Object Landmark Detection
• Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization
• LiStereo: Generate Dense Depth Maps from LIDAR and Stereo Imagery
• Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
• SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation
• DSIC: Deep Stereo Image Compression
• StereoGAN: Bridging Synthetic-to-Real Domain Gap by Joint Optimization of Domain Translation and Stereo Matching
• NavigationNet: A Large-scale Interactive Indoor Navigation Dataset
• DeepTAM: Deep Tracking and Mapping
• MTStereo 2.0: improved accuracy of stereo depth estimation withMax-trees
• Toward 3D Object Reconstruction from Stereo Images
• ADCPNet: Adaptive Disparity Candidates Prediction Network for Efficient Real-Time Stereo Matching
• Learning Stereo Matchability in Disparity Regression Networks
• A Decomposition Model for Stereo Matching
• EDNet: Efficient Disparity Estimation with Cost Volume Combination and Attention-based Spatial Residual
• Meshlet Priors for 3D Mesh Reconstruction
• Self-Supervised Monocular Image Depth Learning and Confidence Estimation
• Learning Optical Flow from a Few Matches
• Deep Learning Stereo Vision at the edge
• Multi-scale Cross-form Pyramid Network for Stereo Matching
• Stereo Object Matching Network
• A-TVSNet: Aggregated Two-View Stereo Network for Multi-View Stereo Depth Estimation
• Volumetric Propagation Network: Stereo-LiDAR Fusion for Long-Range Depth Estimation
• Noise-Sampling Cross Entropy Loss: Improving Disparity Regression Via Cost Volume Aware Regularizer
• Towards Adversarially Robust and Domain Generalizable Stereo Matching by Rethinking DNN Feature Backbones
• Increased-Range Unsupervised Monocular Depth Estimation
• Dedge-AGMNet:an effective stereo matching network optimized by depth edge auxiliary task
• Geometry-Aware Unsupervised Domain Adaptation for Stereo Matching
• VOLDOR: Visual Odometry from Log-logistic Dense Optical flow Residuals
• SCV-Stereo: Learning Stereo Matching from a Sparse Cost Volume
• Video Depth Estimation by Fusing Flow-to-Depth Proposals
• Depth by Poking: Learning to Estimate Depth from Self-Supervised Grasping
• Learning Parallax Attention for Stereo Image Super-Resolution
• Semi-Supervised Disparity Estimation with Deep Feature Reconstruction
• Do End-to-end Stereo Algorithms Under-utilize Information?
• Multi-Scale Cost Volumes Cascade Network for Stereo Matching
• Semi-synthesis: A fast way to produce effective datasets for stereo matching
• PlantStereo: A Stereo Matching Benchmark for Plant Surface Dense Reconstruction