TAP: Text-Aware Pre-training for Text-VQA and Text-Caption

Abstract

In this paper, we propose Text-Aware Pre-training (TAP) for Text-VQA and Text-Caption tasks. These two tasks aim at reading and understanding scene text in images for question answering and image caption generation, respectively. In contrast to the conventional vision-language pre-training that fails to capture scene text and its relationship with the visual and text modalities, TAP explicitly incorporates scene text (generated from OCR engines) in pre-training. With three pre-training tasks, including masked language modeling (MLM), image-text (contrastive) matching (ITM), and relative (spatial) position prediction (RPP), TAP effectively helps the model learn a better aligned representation among the three modalities: text word, visual object, and scene text. Due to this aligned representation learning, even pre-trained on the same downstream task dataset, TAP already boosts the absolute accuracy on the TextVQA dataset by +5.4%, compared with a non-TAP baseline. To further improve the performance, we build a large-scale dataset based on the Conceptual Caption dataset, named OCR-CC, which contains 1.4 million scene text-related image-text pairs. Pre-trained on this OCR-CC dataset, our approach outperforms the state of the art by large margins on multiple tasks, i.e., +8.3% accuracy on TextVQA, +8.6% accuracy on ST-VQA, and +10.2 CIDEr score on TextCaps.

References in corpus (11)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Rosetta: Large scale system for text detection and recognition in images
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images
• UNITER: UNiversal Image-TExt Representation Learning
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Structured Multimodal Attentions for TextVQA
• VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
• Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering

Cited by in corpus (7)

• Learning Transferable Visual Models From Natural Language Supervision
• PAM: Understanding Product Images in Cross Product Category Attribute Extraction
• Scaling Up Vision-Language Pre-training for Image Captioning
• Structured Multimodal Attentions for TextVQA
• SAT: 2D Semantics Assisted Training for 3D Visual Grounding
• Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling
• NarrationBot and InfoBot: A Hybrid System for Automated Video Description