A unified representation network for segmentation with missing modalities

Abstract

Over the last few years machine learning has demonstrated groundbreaking results in many areas of medical image analysis, including segmentation. A key assumption, however, is that the train- and test distributions match. We study a realistic scenario where this assumption is clearly violated, namely segmentation with missing input modalities. We describe two neural network approaches that can handle a variable number of input modalities. The first is modality dropout: a simple but surprisingly effective modification of the training. The second is the unified representation network: a network architecture that maps a variable number of input modalities into a unified representation that can be used for downstream tasks such as segmentation. We demonstrate that modality dropout makes a standard segmentation network reasonably robust to missing modalities, but that the same network works even better if trained on the unified representation.

References in corpus (3)

• A Survey on Deep Learning in Medical Image Analysis
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Analyzing Inverse Problems with Invertible Neural Networks

Cited by in corpus (4)

• Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities
• Feature-enhanced Generation and Multi-modality Fusion based Deep Neural Network for Brain Tumor Segmentation with Missing MR Modalities
• Unified Multi-Modal Image Synthesis for Missing Modality Imputation
• Conditional generator and multi-sourcecorrelation guided brain tumor segmentation with missing MR modalities