Benchmarking Detection Transfer Learning with Vision Transformers

Abstract

Object detection is a central downstream task used to test if pre-trained network parameters confer benefits, such as improved accuracy or training speed. The complexity of object detection methods can make this benchmarking non-trivial when new architectures, such as Vision Transformer (ViT) models, arrive. These difficulties (e.g., architectural incompatibility, slow training, high memory consumption, unknown training formulae, etc.) have prevented recent studies from benchmarking detection transfer learning with standard ViT models. In this paper, we present training techniques that overcome these challenges, enabling the use of standard ViT models as the backbone of Mask R-CNN. These tools facilitate the primary goal of our study: we compare five ViT initializations, including recent state-of-the-art self-supervised learning methods, supervised initialization, and a strong random initialization baseline. Our results show that recent masking-based unsupervised learning methods may, for the first time, provide convincing transfer learning improvements on COCO, increasing box AP up to 4% (absolute) over supervised and prior self-supervised pre-training methods. Moreover, these masking-based initializations scale better, with the improvement growing as model size increases.

References in corpus (6)

• Zero-Shot Text-to-Image Generation
• BEiT: BERT Pre-Training of Image Transformers
• XCiT: Cross-Covariance Image Transformers
• Masked Autoencoders Are Scalable Vision Learners
• Toward Transformer-Based Object Detection
• Multiscale Vision Transformers

Cited by in corpus (6)

• Masked Autoencoders Are Scalable Vision Learners
• Contrastive Masked Autoencoders are Stronger Vision Learners
• CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding
• Towards Phytoplankton Parasite Detection Using Autoencoders
• FashionFail: Addressing Failure Cases in Fashion Object Detection and Segmentation
• Instance-Level Safety-Aware Fidelity of Synthetic Data and Its Calibration