Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations

Abstract

We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available \url{https://github.com/ReubenDo/MHVAE}.

arXiv commentAccepted at MICCAI 2023

References in corpus (5)

• ResViT: Residual vision transformers for multi-modal medical image synthesis
• Disentangled Representation Learning in Cardiac Image Analysis
• Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation
• Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models
• Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets

Cited by in corpus (2)

• Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges
• Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts