PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition

Abstract

3D object recognition has attracted wide research attention in the field of multimedia and computer vision. With the recent proliferation of deep learning, various deep models with different representations have achieved the state-of-the-art performance. Among them, point cloud and multi-view based 3D shape representations are promising recently, and their corresponding deep models have shown significant performance on 3D shape recognition. However, there is little effort concentrating point cloud data and multi-view data for 3D shape representation, which is, in our consideration, beneficial and compensated to each other. In this paper, we propose the Point-View Network (PVNet), the first framework integrating both the point cloud and the multi-view data towards joint 3D shape recognition. More specifically, an embedding attention fusion scheme is proposed that could employ high-level features from the multi-view data to model the intrinsic correlation and discriminability of different structure features from the point cloud data. In particular, the discriminative descriptions are quantified and leveraged as the soft attention mask to further refine the structure feature of the 3D shape. We have evaluated the proposed method on the ModelNet40 dataset for 3D shape classification and retrieval tasks. Experimental results and comparisons with state-of-the-art methods demonstrate that our framework can achieve superior performance.

arXiv commentACM Multimedia Conference 2018

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• Residual Attention Network for Image Classification
• Dynamic Graph CNN for Learning on Point Clouds
• Multi-view Convolutional Neural Networks for 3D Shape Recognition
• FusionNet: 3D Object Classification Using Multiple Data Representations
• PointCNN: Convolution On $\mathcal{X}$-Transformed Points
• Mining Point Cloud Local Structures by Kernel Correlation and Graph Pooling
• Question-Guided Hybrid Convolution for Visual Question Answering

Cited by in corpus (7)

• L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention
• PyramNet: Point Cloud Pyramid Attention Network and Graph Embedding Module for Classification and Segmentation
• PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention
• MeshNet: Mesh Neural Network for 3D Shape Representation
• Attention Models for Point Clouds in Deep Learning: A Survey
• PVRNet: Point-View Relation Neural Network for 3D Shape Recognition
• Exploit Clues from Views: Self-Supervised and Regularized Learning for Multiview Object Recognition