TransformerFusion: Monocular RGB Scene Reconstruction using Transformers

Abstract

We introduce TransformerFusion, a transformer-based 3D scene reconstruction approach. From an input monocular RGB video, the video frames are processed by a transformer network that fuses the observations into a volumetric feature grid representing the scene; this feature grid is then decoded into an implicit 3D scene representation. Key to our approach is the transformer architecture that enables the network to learn to attend to the most relevant image frames for each 3D location in the scene, supervised only by the scene reconstruction task. Features are fused in a coarse-to-fine fashion, storing fine-level features only where needed, requiring lower memory storage and enabling fusion at interactive rates. The feature grid is then decoded to a higher-resolution scene reconstruction, using an MLP-based surface occupancy prediction from interpolated coarse-to-fine 3D features. Our approach results in an accurate surface reconstruction, outperforming state-of-the-art multi-view stereo depth estimation methods, fully-convolutional 3D reconstruction approaches, and approaches using LSTM- or GRU-based recurrent networks for video sequence fusion.

arXiv commentVideo: https://youtu.be/LIpTKYfKSqw

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Transformers in Vision: A Survey
• Language Models are Few-Shot Learners
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction

Cited by in corpus (6)

• Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey
• NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising
• HRBF-Fusion: Accurate 3D reconstruction from RGB-D data using on-the-fly implicits
• Incremental Dense Reconstruction from Monocular Video with Guided Sparse Feature Volume Fusion
• HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction
• VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction