UP-DETR: Unsupervised Pre-training for Object Detection with Transformers

Abstract

DEtection TRansformer (DETR) for object detection reaches competitive performance compared with Faster R-CNN via a transformer encoder-decoder architecture. However, trained with scratch transformers, DETR needs large-scale training data and an extreme long training schedule even on COCO dataset. Inspired by the great success of pre-training transformers in natural language processing, we propose a novel pretext task named random query patch detection in Unsupervised Pre-training DETR (UP-DETR). Specifically, we randomly crop patches from the given image and then feed them as queries to the decoder. The model is pre-trained to detect these query patches from the input image. During the pre-training, we address two critical issues: multi-task learning and multi-query localization. (1) To trade off classification and localization preferences in the pretext task, we find that freezing the CNN backbone is the prerequisite for the success of pre-training transformers. (2) To perform multi-query localization, we develop UP-DETR with multi-query patch detection with attention mask. Besides, UP-DETR also provides a unified perspective for fine-tuning object detection and one-shot detection tasks. In our experiments, UP-DETR significantly boosts the performance of DETR with faster convergence and higher average precision on object detection, one-shot detection and panoptic segmentation. Code and pre-training models: https://github.com/dddzg/up-detr.

Accepted by TPAMI 2022 and CVPR 2021

References in corpus (11)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Transformer in Transformer
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• One-Shot Object Detection with Co-Attention and Co-Excitation
• Parametric Instance Classification for Unsupervised Visual Feature Learning
• One-Shot Object Detection without Fine-Tuning
• Comparison Network for One-Shot Conditional Object Detection