A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation

Abstract

Recent work has shown that optical flow estimation can be formulated as a supervised learning task and can be successfully solved with convolutional networks. Training of the so-called FlowNet was enabled by a large synthetically generated dataset. The present paper extends the concept of optical flow estimation via convolutional networks to disparity and scene flow estimation. To this end, we propose three synthetic stereo video datasets with sufficient realism, variation, and size to successfully train large networks. Our datasets are the first large-scale datasets to enable training and evaluating scene flow methods. Besides the datasets, we present a convolutional network for real-time disparity estimation that provides state-of-the-art results. By combining a flow and disparity estimation network and training it jointly, we demonstrate the first scene flow estimation with a convolutional network.

arXiv commentIncludes supplementary material

Cited by in corpus (88)

• Deep learning in remote sensing: a review
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• A Survey on Deep Learning Techniques for Stereo-based Depth Estimation
• Monocular Depth Estimation Based On Deep Learning: An Overview
• Recursive Cascaded Networks for Unsupervised Medical Image Registration
• Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems
• What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
• MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask
• Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networks
• Perception and Navigation in Autonomous Systems in the Era of Learning: A Survey
• A survey of synthetic data augmentation methods in computer vision
• Synthetic Datasets for Autonomous Driving: A Survey
• Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration
• Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes
• Hierarchical Attention Learning of Scene Flow in 3D Point Clouds
• Masked GANs for Unsupervised Depth and Pose Prediction with Scale Consistency
• DispSegNet: Leveraging Semantics for End-to-End Learning of Disparity Estimation from Stereo Imagery
• Dense Continuous-Time Optical Flow from Events and Frames
• SweepNet: Wide-baseline Omnidirectional Depth Estimation
• StereoSpike: Depth Learning with a Spiking Neural Network
• Self-Supervised Scene Flow Estimation with 4-D Automotive Radar
• A Survey on RGB-D Datasets
• Multiway Non-rigid Point Cloud Registration via Learned Functional Map Synchronization
• Neural Markov Random Field for Stereo Matching
• WHU-Stereo: A Challenging Benchmark for Stereo Matching of High-Resolution Satellite Images
• MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation
• S${}^3$M-Net: Joint Learning of Semantic Segmentation and Stereo Matching for Autonomous Driving
• ASV: Accelerated Stereo Vision System
• When Deep Learning Meets Data Alignment: A Review on Deep Registration Networks (DRNs)
• Motion-based Object Segmentation based on Dense RGB-D Scene Flow
• Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translation
• Topology-Aware Non-Rigid Point Cloud Registration
• Fast Multi-frame Stereo Scene Flow with Motion Segmentation
• Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methods
• DDCNet: Deep Dilated Convolutional Neural Network for Dense Prediction
• Sparse-to-Continuous: Enhancing Monocular Depth Estimation using Occupancy Maps
• EM-driven unsupervised learning for efficient motion segmentation
• Detecting and Identifying Optical Signal Attacks on Autonomous Driving Systems
• Uncertainty Quantification Metrics for Deep Regression
• DCVSMNet: Double Cost Volume Stereo Matching Network
• Depth-aware Blending of Smoothed Images for Bokeh Effect Generation
• A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection
• DurLAR: A High-fidelity 128-channel LiDAR Dataset with Panoramic Ambient and Reflectivity Imagery for Multi-modal Autonomous Driving Applications
• Hybrid Skip: A Biologically Inspired Skip Connection for the UNet Architecture
• SceneFlowFields++: Multi-frame Matching, Visibility Prediction, and Robust Interpolation for Scene Flow Estimation
• Multi-Scale RAFT: Combining Hierarchical Concepts for Learning-based Optical FLow Estimation
• SplatFlow: Learning Multi-frame Optical Flow via Splatting
• Unsupervised Learning of 3D Scene Flow from Monocular Camera
• Lagrangian Motion Magnification with Double Sparse Optical Flow Decomposition
• Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation
• Unsupervised Learning of Depth and Ego-Motion from Cylindrical Panoramic Video
• UAMD-Net: A Unified Adaptive Multimodal Neural Network for Dense Depth Completion
• Sparse Representations for Object and Ego-motion Estimation in Dynamic Scenes
• UnbiasedNets: A Dataset Diversification Framework for Robustness Bias Alleviation in Neural Networks
• Graph-Based Depth Denoising & Dequantization for Point Cloud Enhancement
• Disparity-based Stereo Image Compression with Aligned Cross-View Priors
• Boosting Real-Time Driving Scene Parsing with Shared Semantics
• Transformer-based stereo-aware 3D object detection from binocular images
• Freespace Optical Flow Modeling for Automated Driving
• How Do Neural Networks Estimate Optical Flow? A Neuropsychology-Inspired Study
• Multitask Learning in Minimally Invasive Surgical Vision: A Review
• End-to-end depth from motion with stabilized monocular videos
• Unaligned RGB Guided Hyperspectral Image Super-Resolution with Spatial-Spectral Concordance
• A Lightweight Neural Network for Monocular View Generation with Occlusion Handling
• On Robust Cross-View Consistency in Self-Supervised Monocular Depth Estimation
• SceneEDNet: A Deep Learning Approach for Scene Flow Estimation
• PixSelect: Less but Reliable Pixels for Accurate and Efficient Localization
• TW-SMNet: Deep Multitask Learning of Tele-Wide Stereo Matching
• MaDis-Stereo: Enhanced Stereo Matching via Distilled Masked Image Modeling
• Bent & Broken Bicycles: Leveraging synthetic data for damaged object re-identification
• LeanStereo: A Leaner Backbone based Stereo Network
• RMS-FlowNet++: Efficient and Robust Multi-Scale Scene Flow Estimation for Large-Scale Point Clouds
• An evaluation of Deep Learning based stereo dense matching dataset shift from aerial images and a large scale stereo dataset
• FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model
• 3D Scene Understanding at Urban Intersection using Stereo Vision and Digital Map
• Fast Edge-Aware Occlusion Detection in the Context of Multispectral Camera Arrays
• Document Domain Randomization for Deep Learning Document Layout Extraction
• Robust Shape from Focus via Multiscale Directional Dilated Laplacian and Recurrent Network
• Learning Depth With Very Sparse Supervision
• DDCNet-Multires: Effective Receptive Field Guided Multiresolution CNN for Dense Prediction
• Icy Moon Surface Simulation and Stereo Depth Estimation for Sampling Autonomy
• Color Agnostic Cross-Spectral Disparity Estimation
• Vision-based Lifting of 2D Object Detections for Automated Driving
• Distilling Stereo Networks for Performant and Efficient Leaner Networks
• Real-Time Human Action Recognition on Embedded Platforms
• Disparity Estimation of Planar Reflective Surfaces Using Specular Reflections From a Single Light Source
• FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation
• On the Real-World Generalisability of Optical Flow Models