Toward Transformer-Based Object Detection

Abstract

Transformers have become the dominant model in natural language processing, owing to their ability to pretrain on massive amounts of data, then transfer to smaller, more specific tasks via fine-tuning. The Vision Transformer was the first major attempt to apply a pure transformer model directly to images as input, demonstrating that as compared to convolutional networks, transformer-based architectures can achieve competitive results on benchmark classification tasks. However, the computational complexity of the attention operator means that we are limited to low-resolution inputs. For more complex tasks such as detection or segmentation, maintaining a high input resolution is crucial to ensure that models can properly identify and reflect fine details in their output. This naturally raises the question of whether or not transformer-based architectures such as the Vision Transformer are capable of performing tasks other than classification. In this paper, we determine that Vision Transformers can be used as a backbone by a common detection task head to produce competitive COCO results. The model that we propose, ViT-FRCNN, demonstrates several known properties associated with transformers, including large pretraining capacity and fast fine-tuning performance. We also investigate improvements over a standard detection backbone, including superior performance on out-of-domain images, better performance on large objects, and a lessened reliance on non-maximum suppression. We view ViT-FRCNN as an important stepping stone toward a pure-transformer solution of complex vision tasks such as object detection.

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• Decoupled Weight Decay Regularization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Feature Pyramid Networks for Object Detection
• Billion-scale semi-supervised learning for image classification
• HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
• Curriculum Pre-training for End-to-End Speech Translation
• ObjectNet Dataset: Reanalysis and Correction

Cited by in corpus (16)

• A Survey on Visual Transformer
• Transformer in Transformer
• Centralized Feature Pyramid for Object Detection
• Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion
• You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection
• TransReID: Transformer-based Object Re-Identification
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• ViTGAN: Training GANs with Vision Transformers
• Benchmarking Detection Transfer Learning with Vision Transformers
• Multiscale Vision Transformers
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• CB-HVTNet: A channel-boosted hybrid vision transformer network for lymphocyte assessment in histopathological images
• TRIG: Transformer-Based Text Recognizer with Initial Embedding Guidance
• Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
• CT-block: a novel local and global features extractor for point cloud
• Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One