Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts

Abstract

We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging.

arXiv commentAccepted in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

References in corpus (7)

• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining
• ResViT: Residual vision transformers for multi-modal medical image synthesis
• Disentangled Representation Learning in Cardiac Image Analysis
• Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation
• Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets
• Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations