Probing Image-Language Transformers for Verb Understanding
arXiv:2106.09141
Abstract
Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning (e.g., visual question answering and image retrieval). We are interested in shedding light on the quality of their pretrained representations -- in particular, if these models can distinguish different types of verbs or if they rely solely on nouns in a given sentence. To do so, we collect a dataset of image-sentence pairs (in English) consisting of 421 verbs that are either visual or commonly found in the pretraining data (i.e., the Conceptual Captions dataset). We use this dataset to evaluate pretrained image-language transformers and find that they fail more in situations that require verb understanding compared to other parts of speech. We also investigate what category of verbs are particularly challenging.
References in corpus (7)
- Distributed Representations of Words and Phrases and their Compositionality
- Microsoft COCO Captions: Data Collection and Evaluation Server
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Visual Semantic Role Labeling
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Are we pretraining it right? Digging deeper into visio-linguistic pretraining
- Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers