Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One

Abstract

Depth estimation attracts widespread attention in the computer vision community. However, it is still quite difficult to recover an accurate depth map using only one RGB image. We observe a phenomenon that existing methods tend to fail in different cases, caused by differences in network architecture, loss function and so on. In this work, we investigate into the phenomenon and propose to integrate the strengths of multiple weak depth predictor to build a comprehensive and accurate depth predictor, which is critical for many real-world applications, e.g., 3D reconstruction. Specifically, we construct multiple base (weak) depth predictors by utilizing different Transformer-based and convolutional neural network (CNN)-based architectures. Transformer establishes long-range correlation while CNN preserves local information ignored by Transformer due to the spatial inductive bias. Therefore, the coupling of Transformer and CNN contributes to the generation of complementary depth estimates, which are essential to achieve a comprehensive depth predictor. Then, we design mixers to learn from multiple weak predictions and adaptively fuse them into a strong depth estimate. The resultant model, which we refer to as Transformer-assisted depth ensembles (TEDepth). On the standard NYU-Depth-v2 and KITTI datasets, we thoroughly explore how the neural ensembles affect the depth estimation and demonstrate that our TEDepth achieves better results than previous state-of-the-art approaches. To validate the generalizability across cameras, we directly apply the models trained on NYU-Depth-v2 to the SUN RGB-D dataset without any fine-tuning, and the superior results emphasize its strong generalizability.

arXiv commentAccepted by TMM 2022

References in corpus (19)

• Decoupled Weight Decay Regularization
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• Weighted boxes fusion: Ensembling boxes from different object detection models
• From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• Toward Transformer-Based Object Detection
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• Snapshot Ensembles: Train 1, get M for free
• Improving Adversarial Robustness of Ensembles with Diversity Training
• DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles
• Confidence Calibration for Convolutional Neural Networks Using Structured Dropout
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• VOLO: Vision Outlooker for Visual Recognition
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• Conformer: Local Features Coupling Global Representations for Visual Recognition
• Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction
• Probabilistic Ranking-Aware Ensembles for Enhanced Object Detections