ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
arXiv:2102.03334
Abstract
Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.
arXiv commentICML 2021 Long Presentation
References in corpus (22)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Decoupled Weight Decay Regularization
- Learning Transferable Visual Models From Natural Language Supervision
- mixup: Beyond Empirical Risk Minimization
- Improved Baselines with Momentum Contrastive Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- UNITER: UNiversal Image-TExt Representation Learning
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
- Compositional Attention Networks for Machine Reasoning
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- A Fast Proximal Point Method for Computing Exact Wasserstein Distance
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
- MultiGrain: a unified image embedding for classes and instances
- MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
Cited by in corpus (39)
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
- VLP: A Survey on Vision-Language Pre-training
- FILIP: Fine-grained Interactive Language-Image Pre-Training
- ActionCLIP: A New Paradigm for Video Action Recognition
- How Much Can CLIP Benefit Vision-and-Language Tasks?
- VLT: Vision-Language Transformer and Query Generation for Referring Segmentation
- Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
- Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
- MCare: Learning with Missing Modalities in Multimodal Healthcare Data
- Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentation
- VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
- MERLOT: Multimodal Neural Script Knowledge Models
- Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
- TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation
- ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design
- TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation
- DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Grounding
- Human-Adversarial Visual Question Answering
- Masked Vision-Language Transformer in Fashion
- Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
- Tile Classification Based Viewport Prediction with Multi-modal Fusion Transformer
- Effective End-to-End Vision Language Pretraining with Semantic Visual Loss
- Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
- What Vision-Language Models `See' when they See Scenes
- Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
- Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
- Image-Text Pre-Training for Logo Recognition
- An Explainable Deep Neural Network with Frequency-Aware Channel and Spatial Refinement for Flood Prediction in Sustainable Cities
- Relational Programming with Foundation Models
- Unified Multi-modal Diagnostic Framework with Reconstruction Pre-training and Heterogeneity-combat Tuning
- Is Object Detection Necessary for Human-Object Interaction Recognition?
- Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation
- Integrating Image Features with Convolutional Sequence-to-sequence Network for Multilingual Visual Question Answering
- Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering
- Interpretable Visual Reasoning via Induced Symbolic Space
- Multimodal End-to-End Group Emotion Recognition using Cross-Modal Attention
- Achieving Human Parity on Visual Question Answering
- Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval