Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation

Abstract

We propose a new deep learning method for tumour segmentation when dealing with missing imaging modalities. Instead of producing one network for each possible subset of observed modalities or using arithmetic operations to combine feature maps, our hetero-modal variational 3D encoder-decoder independently embeds all observed modalities into a shared latent representation. Missing data and tumour segmentation can be then generated from this embedding. In our scenario, the input is a random subset of modalities. We demonstrate that the optimisation problem can be seen as a mixture sampling. In addition to this, we introduce a new network architecture building upon both the 3D U-Net and the Multi-Modal Variational Auto-Encoder (MVAE). Finally, we evaluate our method on BraTS2018 using subsets of the imaging modalities as input. Our model outperforms the current state-of-the-art method for dealing with missing modalities and achieves similar performance to the subset-specific equivalent networks.

arXiv commentAccepted at MICCAI 2019

References in corpus (5)

• Auto-Encoding Variational Bayes
• NiftyNet: a deep-learning platform for medical imaging
• Ladder Variational Autoencoders
• Multimodal Generative Models for Scalable Weakly-Supervised Learning
• Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions

Cited by in corpus (13)

• Feature-enhanced Generation and Multi-modality Fusion based Deep Neural Network for Brain Tumor Segmentation with Missing MR Modalities
• Unified Multi-Modal Image Synthesis for Missing Modality Imputation
• Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
• Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets
• Brain MRI Tumor Segmentation with Adversarial Networks
• Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy
• Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations
• ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities
• Conditional generator and multi-sourcecorrelation guided brain tumor segmentation with missing MR modalities
• InstantGroup: Instant Template Generation for Scalable Group of Brain MRI Registration
• Bayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registration
• Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
• On the Limitations of Multimodal VAEs