Learning to Reconstruct and Segment 3D Objects

Abstract

To endow machines with the ability to perceive the real-world in a three dimensional representation as we do as humans is a fundamental and long-standing topic in Artificial Intelligence. Given different types of visual inputs such as images or point clouds acquired by 2D/3D sensors, one important goal is to understand the geometric structure and semantics of the 3D environment. Traditional approaches usually leverage hand-crafted features to estimate the shape and semantics of objects or scenes. However, they are difficult to generalize to novel objects and scenarios, and struggle to overcome critical issues caused by visual occlusions. By contrast, we aim to understand scenes and the objects within them by learning general and robust representations using deep neural networks, trained on large-scale real-world 3D data. To achieve these aims, this thesis makes three core contributions from object-level 3D shape estimation from single or multiple views to scene-level semantic understanding.

arXiv commentDPhil (PhD) Thesis 2020, University of Oxford https://ora.ox.ac.uk/objects/uuid:5f9cd30d-0ee7-412d-ba49-44f5fd76bf28

References in corpus (37)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Focal Loss for Dense Object Detection
• Deep Convolutional Inverse Graphics Network
• PCPNET: Learning Local Shape Properties from Raw Point Clouds
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• Learning What and Where to Draw
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction
• Dense 3D Object Reconstruction from a Single Depth View
• Attention-based Pyramid Aggregation Network for Visual Place Recognition
• SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth
• Learning to Infer Implicit Surfaces without 3D Supervision
• MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation
• 3D-BEVIS: Bird's-Eye-View Instance Segmentation
• Learning Permutations with Sinkhorn Policy Gradient
• Stochastic Optimization of Sorting Networks via Continuous Relaxations
• Multiview Aggregation for Learning Category-Specific Shape Reconstruction
• Parallelizing Linear Recurrent Neural Nets Over Sequence Length
• Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspots
• Exploiting Local and Global Structure for Point Cloud Semantic Segmentation with Contextual Point Representations
• Fusing Bird View LIDAR Point Cloud and Front View Camera Image for Deep Object Detection
• SemanticPOSS: A Point Cloud Dataset with Large Quantity of Dynamic Instances
• Single-view Object Shape Reconstruction Using Deep Shape Prior and Silhouette
• Deep Attentional Structured Representation Learning for Visual Recognition
• AdvectiveNet: An Eulerian-Lagrangian Fluidic reservoir for Point Cloud Processing
• Toward 3D Object Reconstruction from Stereo Images
• Learning and Memorizing Representative Prototypes for 3D Point Cloud Semantic and Instance Segmentation