Unified Multi-Modal Image Synthesis for Missing Modality Imputation

Abstract

Multi-modal medical images provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-modal images, thus limiting the usage of multi-modal data for clinical purposes. To address this issue, in this paper, we propose a novel unified multi-modal image synthesis method for missing modality imputation. Our method overall takes a generative adversarial architecture, which aims to synthesize missing modalities from any combination of available ones with a single model. To this end, we specifically design a Commonality- and Discrepancy-Sensitive Encoder for the generator to exploit both modality-invariant and specific information contained in input modalities. The incorporation of both types of information facilitates the generation of images with consistent anatomy and realistic details of the desired distribution. Besides, we propose a Dynamic Feature Unification Module to integrate information from a varying number of available modalities, which enables the network to be robust to random missing modalities. The module performs both hard integration and soft integration, ensuring the effectiveness of feature combination while avoiding information loss. Verified on two public multi-modal magnetic resonance datasets, the proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.

IEEE TMI accepted final version

References in corpus (10)

• Knowledge Distillation: A Survey
• Rethinking the Value of Network Pruning
• ResViT: Residual vision transformers for multi-modal medical image synthesis
• Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities
• Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation
• One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation
• Bi-Modality Medical Image Synthesis Using Semi-Supervised Sequential Generative Adversarial Networks
• A unified representation network for segmentation with missing modalities
• DiffuseExpand: Expanding dataset for 2D medical image segmentation using diffusion models
• SFusion: Self-attention based N-to-One Multimodal Fusion Block

Cited by in corpus (2)

• A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends
• Deep infant brain segmentation from multi-contrast MRI