Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning

Abstract

Masked Autoencoders (MAE) have demonstrated promising performance in self-supervised learning for both 2D and 3D computer vision. Nevertheless, existing MAE-based methods still have certain drawbacks. Firstly, the functional decoupling between the encoder and decoder is incomplete, which limits the encoder's representation learning ability. Secondly, downstream tasks solely utilize the encoder, failing to fully leverage the knowledge acquired through the encoder-decoder architecture in the pre-text task. In this paper, we propose Point Regress AutoEncoder (Point-RAE), a new scheme for regressive autoencoders for point cloud self-supervised learning. The proposed method decouples functions between the decoder and the encoder by introducing a mask regressor, which predicts the masked patch representation from the visible patch representation encoded by the encoder and the decoder reconstructs the target from the predicted masked patch representation. By doing so, we minimize the impact of decoder updates on the representation space of the encoder. Moreover, we introduce an alignment constraint to ensure that the representations for masked patches, predicted from the encoded representations of visible patches, are aligned with the masked patch presentations computed from the encoder. To make full use of the knowledge learned in the pre-training stage, we design a new finetune mode for the proposed Point-RAE. Extensive experiments demonstrate that our approach is efficient during pre-training and generalizes well on various downstream tasks. Specifically, our pre-trained models achieve a high accuracy of \textbf{90.28\%} on the ScanObjectNN hardest split and \textbf{94.1\%} accuracy on ModelNet40, surpassing all the other self-supervised learning methods. Our code and pretrained model are public available at: \url{https://github.com/liuyyy111/Point-RAE}.

References in corpus (18)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• ShapeNet: An Information-Rich 3D Model Repository
• PCT: Point cloud transformer
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• BEiT: BERT Pre-Training of Image Transformers
• PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies
• Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• PRA-Net: Point Relation-Aware Network for 3D Point Cloud Analysis
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
• P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting
• Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining
• PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
• Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?
• Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning
• Masked Surfel Prediction for Self-Supervised Point Cloud Learning