Self-Training Vision Language BERTs with a Unified Conditional Model

Abstract

Natural language BERTs are trained with language corpus in a self-supervised manner. Unlike natural language BERTs, vision language BERTs need paired data to train, which restricts the scale of VL-BERT pretraining. We propose a self-training approach that allows training VL-BERTs from unlabeled image data. The proposed method starts with our unified conditional model -- a vision language BERT model that can perform zero-shot conditional generation. Given different conditions, the unified conditional model can generate captions, dense captions, and even questions. We use the labeled image data to train a teacher model and use the trained model to generate pseudo captions on unlabeled image data. We then combine the labeled data and pseudo labeled data to train a student model. The process is iterated by putting the student model as a new teacher. By using the proposed self-training approach and only 300k unlabeled extra data, we are able to get competitive or even better performances compared to the models of similar model size trained with 3 million extra image data.

References in corpus (7)

• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Rethinking Pre-training and Self-training
• Billion-scale semi-supervised learning for image classification
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning

Cited by in corpus (2)

• SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
• Compositional Kronecker Context Optimization for Vision-Language Models