Self-supervised Modal and View Invariant Feature Learning
arXiv:2005.14169
Abstract
Most of the existing self-supervised feature learning methods for 3D data either learn 3D features from point cloud data or from multi-view images. By exploring the inherent multi-modality attributes of 3D objects, in this paper, we propose to jointly learn modal-invariant and view-invariant features from different modalities including image, point cloud, and mesh with heterogeneous networks for 3D data. In order to learn modal- and view-invariant features, we propose two types of constraints: cross-modal invariance constraint and cross-view invariant constraint. Cross-modal invariance constraint forces the network to maximum the agreement of features from different modalities for same objects, while the cross-view invariance constraint forces the network to maximum agreement of features from different views of images for same objects. The quality of learned features has been tested on different downstream tasks with three modalities of data including point cloud, multi-view images, and mesh. Furthermore, the invariance cross different modalities and views are evaluated with the cross-modal retrieval task. Extensive evaluation results demonstrate that the learned features are robust and have strong generalizability across different tasks.
References in corpus (9)
- A Simple Framework for Contrastive Learning of Visual Representations
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Unsupervised Learning of Video Representations using LSTMs
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Contrastive Multiview Coding
- Watching the World Go By: Representation Learning from Unlabeled Videos
- MortonNet: Self-Supervised Learning of Local Features in 3D Point Clouds
- Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences