Pix2seq: A Language Modeling Framework for Object Detection

Abstract

We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection as a language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed as sequences of discrete tokens, and we train a neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations, our approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms.

arXiv commentICLR'22. Code and pretrained models at https://github.com/google-research/pix2seq

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Sequence to Sequence Learning with Neural Networks
• Decoupled Weight Decay Regularization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Improving neural networks by preventing co-adaptation of feature detectors
• Language Models are Few-Shot Learners
• Zero-Shot Text-to-Image Generation
• The Curious Case of Neural Text Degeneration
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Some Improvements on Deep Convolutional Neural Network Based Image Classification
• Recurrent Neural Networks for Semantic Instance Segmentation
• Learning to decompose for object detection and instance segmentation

Cited by in corpus (5)

• Transformers in Vision: A Survey
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• Contrastive Proposal Extension with LSTM Network for Weakly Supervised Object Detection