BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Abstract

Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.

Cited by in corpus (42)

• Towards Small Object Editing: A Benchmark Dataset and A Training-Free Approach
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Search
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• Materials science in the era of large language models: a perspective
• AVscript: Accessible Video Editing with Audio-Visual Scripts
• ZenSVI: An Open-Source Software for the Integrated Acquisition, Processing and Analysis of Street View Imagery Towards Scalable Urban Science
• Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval
• SoccerNet 2023 Challenges Results
• 360-Degree Panorama Generation from Few Unregistered NFoV Images
• Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning
• Epigraphics: Message-Driven Infographics Authoring
• Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separation
• RTQ: Rethinking Video-language Understanding Based on Image-text Model
• Automatic Medical Report Generation: Methods and Applications
• GUing: A Mobile GUI Search Engine using a Vision-Language Model
• MemoVis: A GenAI-Powered Tool for Creating Companion Reference Images for 3D Design Feedback
• Fusing Domain-Specific Content from Large Language Models into Knowledge Graphs for Enhanced Zero Shot Object State Classification
• Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
• MatplotAlt: A Python Library for Adding Alt Text to Matplotlib Figures in Computational Notebooks
• IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training
• Visual Language Models as Operator Agents in the Space Domain
• Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
• AttriPrompter: Auto-Prompting with Attribute Semantics for Zero-shot Nuclei Detection via Visual-Language Pre-trained Models
• Multimedia Generative Script Learning for Task Planning
• PEAR: Phrase-Based Hand-Object Interaction Anticipation
• Automated Description Generation of Cytologic Findings for Lung Cytological Images Using a Pretrained Vision Model and Dual Text Decoders: Preliminary Study
• Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation
• AI as a Tool for Fair Journalism: Case Studies from Malta
• Language Guided Adversarial Purification
• AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimization
• Leveraging Self-Supervised Vision Transformers for Segmentation-based Transfer Function Design
• Intelligent Anomaly Detection for Lane Rendering Using Transformer with Self-Supervised Pre-Training and Customized Fine-Tuning
• A Systematic Review of Open Datasets Used in Text-to-Image (T2I) Gen AI Model Safety
• HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
• When Better Eyes Lead to Blindness: A Diagnostic Study of the Information Bottleneck in CNN-LSTM Image Captioning Models
• printf: Preference Modeling Based on User Reviews with Item Images and Textual Information via Graph Learning
• Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
• GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models
• DIVE: Towards Descriptive and Diverse Visual Commonsense Generation