Scaling Up Vision-Language Pre-training for Image Captioning

Abstract

In recent years, we have witnessed significant performance boost in the image captioning task based on vision-language pre-training (VLP). Scale is believed to be an important factor for this advance. However, most existing work only focuses on pre-training transformers with moderate sizes (e.g., 12 or 24 layers) on roughly 4 million images. In this paper, we present LEMON, a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning. We use the state-of-the-art VinVL model as our reference model, which consists of an image feature extractor and a transformer model, and scale the transformer both up and down, with model sizes ranging from 13 to 675 million parameters. In terms of data, we conduct experiments with up to 200 million image-text pairs which are automatically collected from web based on the alt attribute of the image (dubbed as ALT200M). Extensive analysis helps to characterize the performance trend as the model size and the pre-training data size increase. We also compare different training recipes, especially for training on large-scale noisy data. As a result, LEMON achieves new state of the arts on several major image captioning benchmarks, including COCO Caption, nocaps, and Conceptual Captions. We also show LEMON can generate captions with long-tail visual concepts when used in a zero-shot manner.

References in corpus (16)

• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scaling Laws for Neural Language Models
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• Unifying Vision-and-Language Tasks via Text Generation
• Scaling Laws for Autoregressive Generative Modeling
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• MiniVLM: A Smaller and Faster Vision-Language Model
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption