FroDO: From Detections to 3D Objects

Abstract

Object-oriented maps are important for scene understanding since they jointly capture geometry and semantics, allow individual instantiation and meaningful reasoning about objects. We introduce FroDO, a method for accurate 3D reconstruction of object instances from RGB video that infers object location, pose and shape in a coarse-to-fine manner. Key to FroDO is to embed object shapes in a novel learnt space that allows seamless switching between sparse point cloud and dense DeepSDF decoding. Given an input sequence of localized RGB frames, FroDO first aggregates 2D detections to instantiate a category-aware 3D bounding box per object. A shape code is regressed using an encoder network before optimizing shape and pose further under the learnt shape priors using sparse and dense shape representations. The optimization uses multi-view geometric, photometric and silhouette losses. We evaluate on real-world datasets, including Pix3D, Redwood-OS, and ScanNet, for single-view, multi-view, and multi-object reconstruction.

To be published in CVPR 2020. The first two authors contributed equally

References in corpus (4)

• A Large Dataset of Object Scans
• Controlling Neural Level Sets
• Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
• Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction

Cited by in corpus (5)

• ELLIPSDF: Joint Object Pose and Shape Optimization with a Bi-level Ellipsoid and Signed Distance Function Description
• RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction
• Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image
• Single-View 3D Object Reconstruction from Shape Priors in Memory
• MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videos