Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

Abstract

Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL), which adopts a BERT-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performance for both vision-language understanding tasks (diagnosis classification, medical image-report retrieval, medical visual question answering) and vision-language generation task (radiology report generation). By statistically and rigorously evaluating the proposed model on four downstream tasks with three radiographic image-report datasets (MIMIC-CXR, Open-I, and VQA-RAD), we empirically demonstrate the superior downstream task performance of MedViLL against various baselines, including task-specific architectures. The source code is publicly available at: https://github.com/SuperSupermoon/MedViLL

Accepted by IEEE Journal of Biomedical and Health Informatics

References in corpus (7)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Publicly Available Clinical BERT Embeddings
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• A Multimodal Memes Classification: A Survey and Open Research Issues
• X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
• EMIXER: End-to-end Multimodal X-ray Generation via Self-supervision

Cited by in corpus (11)

• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• CLIP in Medical Imaging: A Survey
• Automated Radiology Report Generation: A Review of Recent Advances
• LLM-driven Multimodal Target Volume Contouring in Radiation Oncology
• A scoping review on multimodal deep learning in biomedical images and texts
• CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation
• From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine
• Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge
• Local Contrastive Learning for Medical Image Recognition
• Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports
• Unified Multi-modal Diagnostic Framework with Reconstruction Pre-training and Heterogeneity-combat Tuning