Generic 3D Representation via Pose Estimation and Matching

Abstract

Though a large body of computer vision research has investigated developing generic semantic representations, efforts towards developing a similar representation for 3D has been limited. In this paper, we learn a generic 3D representation through solving a set of foundational proxy 3D tasks: object-centric camera pose estimation and wide baseline feature matching. Our method is based upon the premise that by providing supervision over a set of carefully selected foundational tasks, generalization to novel tasks and abstraction capabilities can be achieved. We empirically show that the internal representation of a multi-task ConvNet trained to solve the above core problems generalizes to novel 3D tasks (e.g., scene layout estimation, object pose estimation, surface normal estimation) without the need for fine-tuning and shows traits of abstraction abilities (e.g., cross-modality pose estimation). In the context of the core supervised tasks, we demonstrate our representation achieves state-of-the-art wide baseline feature matching results without requiring apriori rectification (unlike SIFT and the majority of learned features). We also show 6DOF camera pose estimation given a pair local image patches. The accuracy of both supervised tasks come comparable to humans. Finally, we contribute a large-scale dataset composed of object-centric street view scenes along with point correspondences and camera pose information, and conclude with a discussion on the learned representation and open research questions.

Published in ECCV16. See the project website http://3drepresentation.stanford.edu/ and dataset website https://github.com/amir32002/3D_Street_View

References in corpus (9)

• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• One weird trick for parallelizing convolutional neural networks
• Computing the Stereo Matching Cost with a Convolutional Neural Network
• Descriptor Matching with Convolutional Neural Networks: a Comparison to SIFT
• Learning to Compare Image Patches via Convolutional Neural Networks
• Unsupervised Learning of Visual Representations using Videos
• PN-Net: Conjoined Triple Deep Network for Learning Local Image Descriptors
• Learning to See by Moving

Cited by in corpus (20)

• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Survey
• Self-supervised learning of a facial attribute embedding from video
• Time-Contrastive Networks: Self-Supervised Learning from Video
• OctNet: Learning Deep 3D Representations at High Resolutions
• When Regression Meets Manifold Learning for Object Recognition and Pose Estimation
• TorontoCity: Seeing the World with a Million Eyes
• Learning models for visual 3D localization with implicit mapping
• RCNet: Deep Recurrent Collaborative Network for Multi-View Low-Light Image Enhancement
• ACNe: Attentive Context Normalization for Robust Permutation-Equivariant Learning
• Feature Pyramid Network for Multi-task Affective Analysis
• Eigendecomposition-free Training of Deep Networks with Zero Eigenvalue-based Losses
• IV-SLAM: Introspective Vision for Simultaneous Localization and Mapping
• On Machine Learning and Structure for Mobile Robots
• Learning Multiple Dense Prediction Tasks from Partially Annotated Data
• Pre-training Attention Mechanisms
• Metric-based Regularization and Temporal Ensemble for Multi-task Learning using Heterogeneous Unsupervised Tasks
• Knowledge as Invariance -- History and Perspectives of Knowledge-augmented Machine Learning
• Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problems
• Equivariant Contrastive Learning