Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine

Abstract

Pixel-level 2D object semantic understanding is an important topic in computer vision and could help machine deeply understand objects (e.g. functionality and affordance) in our daily life. However, most previous methods directly train on correspondences in 2D images, which is end-to-end but loses plenty of information in 3D spaces. In this paper, we propose a new method on predicting image corresponding semantics in 3D domain and then projecting them back onto 2D images to achieve pixel-level understanding. In order to obtain reliable 3D semantic labels that are absent in current image datasets, we build a large scale keypoint knowledge engine called KeypointNet, which contains 103,450 keypoints and 8,234 3D models from 16 object categories. Our method leverages the advantages in 3D vision and can explicitly reason about objects self-occlusion and visibility. We show that our method gives comparative and even superior results on standard semantic benchmarks.

Accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence; To appear in upcoming issues

References in corpus (14)

• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
• Spherical CNNs
• Neighbourhood Consensus Networks
• Learning to Infer Implicit Surfaces without 3D Supervision
• Learning to Infer and Execute 3D Shape Programs
• SPair-71k: A Large-scale Benchmark for Semantic Correspondence
• SDM-NET: Deep Generative Network for Structured Deformable Mesh
• KeypointNet: A Large-scale 3D Keypoint Dataset Aggregated from Numerous Human Annotations
• Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation
• Front2Back: Single View 3D Shape Reconstruction via Front to Back Prediction