Three Pillars improving Vision Foundation Model Distillation for Lidar

Abstract

Self-supervised image backbones can be used to address complex 2D tasks (e.g., semantic segmentation, object discovery) very efficiently and with little or no downstream supervision. Ideally, 3D backbones for lidar should be able to inherit these properties after distillation of these powerful 2D features. The most recent methods for image-to-lidar distillation on autonomous driving data show promising results, obtained thanks to distillation methods that keep improving. Yet, we still notice a large performance gap when measuring the quality of distilled and fully supervised features by linear probing. In this work, instead of focusing only on the distillation method, we study the effect of three pillars for distillation: the 3D backbone, the pretrained 2D backbones, and the pretraining dataset. In particular, thanks to our scalable distillation method named ScaLR, we show that scaling the 2D and 3D backbones and pretraining on diverse datasets leads to a substantial improvement of the feature quality. This allows us to significantly reduce the gap between the quality of distilled and fully-supervised 3D features, and to improve the robustness of the pretrained backbones to domain gaps and perturbations.

arXiv commentThe code is available at https://github.com/valeoai/ScaLR

References in corpus (21)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• DINOv2: Learning Robust Visual Features without Supervision
• BEiT: BERT Pre-Training of Image Transformers
• iBOT: Image BERT Pre-Training with Online Tokenizer
• A Cookbook of Self-Supervised Learning
• Segment Everything Everywhere All at Once
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
• Localizing Objects with Self-Supervised Transformers and no Labels
• Masked Autoencoder for Self-Supervised Pre-training on Lidar Point Clouds
• Segment Any Point Cloud Sequences by Distilling Vision Foundation Models
• GFNet: Geometric Flow Network for 3D Point Cloud Semantic Segmentation
• SALUDA: Surface-based Automotive Lidar Unsupervised Domain Adaptation
• Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?
• PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving
• Rethinking Evaluation Protocols of Visual Representations Learned via Self-supervised Learning
• Robo3D: Towards Robust and Reliable 3D Perception against Corruptions
• Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupancy Autoencoders