Transformers in Unsupervised Structure-from-Motion

Abstract

Transformers have revolutionized deep learning based computer vision with improved performance as well as robustness to natural corruptions and adversarial attacks. Transformers are used predominantly for 2D vision tasks, including image classification, semantic segmentation, and object detection. However, robots and advanced driver assistance systems also require 3D scene understanding for decision making by extracting structure-from-motion (SfM). We propose a robust transformer-based monocular SfM method that learns to predict monocular pixel-wise depth, ego vehicle's translation and rotation, as well as camera's focal length and principal point, simultaneously. With experiments on KITTI and DDAD datasets, we demonstrate how to adapt different vision transformers and compare them against contemporary CNN-based methods. Our study shows that transformer-based architecture, though lower in run-time efficiency, achieves comparable performance while being more robust against natural corruptions, as well as untargeted and targeted attacks.

International Joint Conference on Computer Vision, Imaging and Computer Graphics. Cham: Springer Nature Switzerland, 2022. Published at "Communications in Computer and Information Science, vol 1815. Springer Nature". arXiv admin note: text overlap with arXiv:2202.03131

References in corpus (5)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• PVT v2: Improved Baselines with Pyramid Vision Transformer
• Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks
• On the interplay of adversarial robustness and architecture components: patches, convolution and attention