IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training

Abstract

Vision-Language Pre-training (VLP) with large-scale image-text pairs has demonstrated superior performance in various fields. However, the image-text pairs co-occurrent on the Internet typically lack explicit alignment information, which is suboptimal for VLP. Existing methods proposed to adopt an off-the-shelf object detector to utilize additional image tag information. However, the object detector is time-consuming and can only identify the pre-defined object categories, limiting the model capacity. Inspired by the observation that the texts incorporate incomplete fine-grained image information, we introduce IDEA, which stands for increasing text diversity via online multi-label recognition for VLP. IDEA shows that multi-label learning with image tags extracted from the texts can be jointly optimized during VLP. Moreover, IDEA can identify valuable image tags online to provide more explicit textual supervision. Comprehensive experiments demonstrate that IDEA can significantly boost the performance on multiple downstream datasets with a small extra computational cost.

arXiv commentAccepted by the 30th ACM International Conference on Multimedia (ACM MM 2022)

References in corpus (19)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Representation Learning with Contrastive Predictive Coding
• Learning Transferable Visual Models From Natural Language Supervision
• NLTK: The Natural Language Toolkit
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
• Query2Label: A Simple Transformer Way to Multi-Label Classification
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• A Survey of Vision-Language Pre-Trained Models
• Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
• Learning a Deep ConvNet for Multi-label Classification with Partial Labels
• Simple and Robust Loss Design for Multi-Label Learning with Missing Labels
• A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision