Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

Abstract

We propose Pixel-BERT to align image pixels with text by deep multi-modal transformers that jointly learn visual and language embedding in a unified end-to-end framework. We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks. Our Pixel-BERT which aligns semantic connection in pixel and text level solves the limitation of task-specific visual representation for vision and language tasks. It also relieves the cost of bounding box annotations and overcomes the unbalance between semantic labels in visual task and language semantic. To provide a better representation for down-stream tasks, we pre-train a universal end-to-end model with image and sentence pairs from Visual Genome dataset and MS-COCO dataset. We propose to use a random pixel sampling mechanism to enhance the robustness of visual representation and to apply the Masked Language Model and Image-Text Matching as pre-training tasks. Extensive experiments on downstream tasks with our pre-trained model show that our approach makes the most state-of-the-arts in downstream tasks, including Visual Question Answering (VQA), image-text retrieval, Natural Language for Visual Reasoning for Real (NLVR). Particularly, we boost the performance of a single model in VQA task by 2.17 points compared with SOTA under fair comparison.

References in corpus (9)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Layer Normalization
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Cited by in corpus (64)

• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• VLP: A Survey on Vision-Language Pre-training
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• ${\mathbf{C}}^2$Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• VLT: Vision-Language Transformer and Query Generation for Referring Segmentation
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• VinVL: Revisiting Visual Representations in Vision-Language Models
• Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
• M6: A Chinese Multimodal Pretrainer
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamese
• A Multimodal Memes Classification: A Survey and Open Research Issues
• RepsNet: Combining Vision with Language for Automated Medical Reports
• MiniVLM: A Smaller and Faster Vision-Language Model
• UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
• X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
• RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering
• Cross-Modal Pre-Aligned Method with Global and Local Information for Remote-Sensing Image and Text Retrieval
• Open-Vocabulary Object Detection Using Captions
• Scaling Up Vision-Language Pre-training for Image Captioning
• Hateful Memes Detection via Complementary Visual and Linguistic Networks
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Effective End-to-End Vision Language Pretraining with Semantic Visual Loss
• AliMe MKG: A Multi-modal Knowledge Graph for Live-streaming E-commerce
• Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
• Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
• SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction
• Video-Text Pre-training with Learned Regions
• A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
• LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
• Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Probing Image-Language Transformers for Verb Understanding
• SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
• ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration
• Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
• Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization
• Understanding Mobile GUI: from Pixel-Words to Screen-Sentences
• A Multi-level Alignment Training Scheme for Video-and-Language Grounding
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Achieving Human Parity on Visual Question Answering
• Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation
• VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition
• Separating Skills and Concepts for Novel Visual Question Answering
• Co-attentional Transformers for Story-Based Video Understanding
• SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval