Unified Multi-modal Diagnostic Framework with Reconstruction Pre-training and Heterogeneity-combat Tuning

Abstract

Medical multi-modal pre-training has revealed promise in computer-aided diagnosis by leveraging large-scale unlabeled datasets. However, existing methods based on masked autoencoders mainly rely on data-level reconstruction tasks, but lack high-level semantic information. Furthermore, two significant heterogeneity challenges hinder the transfer of pre-trained knowledge to downstream tasks, \textit{i.e.}, the distribution heterogeneity between pre-training data and downstream data, and the modality heterogeneity within downstream data. To address these challenges, we propose a Unified Medical Multi-modal Diagnostic (UMD) framework with tailored pre-training and downstream tuning strategies. Specifically, to enhance the representation abilities of vision and language encoders, we propose the Multi-level Reconstruction Pre-training (MR-Pretrain) strategy, including a feature-level and data-level reconstruction, which guides models to capture the semantic information from masked inputs of different modalities. Moreover, to tackle two kinds of heterogeneities during the downstream tuning, we present the heterogeneity-combat downstream tuning strategy, which consists of a Task-oriented Distribution Calibration (TD-Calib) and a Gradient-guided Modality Coordination (GM-Coord). In particular, TD-Calib fine-tunes the pre-trained model regarding the distribution of downstream datasets, and GM-Coord adjusts the gradient weights according to the dynamic optimization status of different modalities. Extensive experiments on five public medical datasets demonstrate the effectiveness of our UMD framework, which remarkably outperforms existing approaches on three kinds of downstream tasks.

to be published in IEEE JBHI; Code available at https://github.com/helenypzhang/UMD

References in corpus (17)

• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• Gaussian Error Linear Units (GELUs)
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Recent advances and clinical applications of deep learning in medical image analysis
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
• Annotation-efficient deep learning for automatic medical image segmentation
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• Test-Time Training with Masked Autoencoders
• Masked Vision and Language Modeling for Multi-modal Representation Learning
• MAR: Masked Autoencoders for Efficient Action Recognition
• DeepMIM: Deep Supervision for Masked Image Modeling
• Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
• FreMIM: Fourier Transform Meets Masked Image Modeling for Medical Image Segmentation