You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection

Abstract

Can Transformer perform 2D object- and region-level recognition from a pure sequence-to-sequence perspective with minimal knowledge about the 2D spatial structure? To answer this question, we present You Only Look at One Sequence (YOLOS), a series of object detection models based on the vanilla Vision Transformer with the fewest possible modifications, region priors, as well as inductive biases of the target task. We find that YOLOS pre-trained on the mid-sized ImageNet-1k dataset only can already achieve quite competitive performance on the challenging COCO object detection benchmark, e.g., YOLOS-Base directly adopted from BERT-Base architecture can obtain 42.0 box AP on COCO val. We also discuss the impacts as well as limitations of current pre-train schemes and model scaling strategies for Transformer in vision through YOLOS. Code and pre-trained models are available at https://github.com/hustvl/YOLOS.

arXiv commentNeurIPS 2021 Camera Ready

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• YOLOv3: An Incremental Improvement
• Learning Transferable Visual Models From Natural Language Supervision
• mixup: Beyond Empirical Risk Minimization
• Gaussian Error Linear Units (GELUs)
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Wide Residual Networks
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• A large annotated corpus for learning natural language inference
• GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
• AutoAssign: Differentiable Label Assignment for Dense Object Detection
• Toward Transformer-Based Object Detection
• Scaled-YOLOv4: Scaling Cross Stage Partial Network
• Learning Deep Transformer Models for Machine Translation
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining

Cited by in corpus (12)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS
• VLP: A Survey on Vision-Language Pre-training
• CSWin-UNet: Transformer UNet with Cross-Shaped Windows for Medical Image Segmentation
• TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting
• Vision Transformers Need Registers
• ViDT: An Efficient and Effective Fully Transformer-based Object Detector
• A Survey of Visual Transformers
• Tracking Instances as Queries
• Guiding Query Position and Performing Similar Attention for Transformer-Based Detection Heads
• MammoMix: Leveraging Mixture of Experts for Robust Mammogram Breast Detection