Multimodal Semi-Supervised Learning for 3D Objects

Abstract

In recent years, semi-supervised learning has been widely explored and shows excellent data efficiency for 2D data. There is an emerging need to improve data efficiency for 3D tasks due to the scarcity of labeled 3D data. This paper explores how the coherence of different modelities of 3D data (e.g. point cloud, image, and mesh) can be used to improve data efficiency for both 3D classification and retrieval tasks. We propose a novel multimodal semi-supervised learning framework by introducing instance-level consistency constraint and a novel multimodal contrastive prototype (M2CP) loss. The instance-level consistency enforces the network to generate consistent representations for multimodal data of the same object regardless of its modality. The M2CP maintains a multimodal prototype for each class and learns features with small intra-class variations by minimizing the feature distance of each object to its prototype while maximizing the distance to the others. Our proposed framework significantly outperforms all the state-of-the-art counterparts for both classification and retrieval tasks by a large margin on the modelNet10 and ModelNet40 datasets.

BMVC 2021 poster

References in corpus (16)

• A Simple Framework for Contrastive Learning of Visual Representations
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• MeshCNN: A Network with an Edge
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• A Simple Semi-Supervised Learning Framework for Object Detection
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• Self-Supervised Few-Shot Learning on Point Clouds
• Weakly Supervised Semantic Point Cloud Segmentation:Towards 10X Fewer Labels
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning
• Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
• Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences
• Pri3D: Can 3D Priors Help 2D Representation Learning?
• Cross-modal Center Loss

Cited by in corpus (2)

• Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation
• Learning-accelerated A* Search for Risk-aware Path Planning