Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network

Abstract

Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image registration and as supplementary training samples to boost the generalization capability of a classifier. Especially, synthesized computed tomography (CT) data can provide X-ray attenuation map for radiation therapy planning. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 3D images using unpaired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distortion in cross-modality synthesis and 3) improving volume segmentation by using synthetic data for modalities with limited training samples. We show that these goals can be achieved with an end-to-end 3D convolutional neural network (CNN) composed of mutually-beneficial generators and segmentors for image synthesis and segmentation tasks. The generators are trained with an adversarial loss, a cycle-consistency loss, and also a shape-consistency loss, which is supervised by segmentors, to reduce the geometric distortion. From the segmentation view, the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors prompt each other alternatively in an end-to-end training fashion. With extensive experiments on a dataset including a total of 4,496 CT and magnetic resonance imaging (MRI) cardiovascular volumes, we show both tasks are beneficial to each other and coupling these two tasks results in better performance than solving them exclusively.

arXiv commentCVPR2018 (a small revision in v2)

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Instance Normalization: The Missing Ingredient for Fast Stylization
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• SegAN: Adversarial Network with Multi-scale $L_1$ Loss for Medical Image Segmentation
• Semantic Segmentation using Adversarial Networks
• Adversarial Networks for the Detection of Aggressive Prostate Cancer
• Towards Adversarial Retinal Image Synthesis
• Medical Image Synthesis with Context-Aware Generative Adversarial Networks
• Unsupervised domain adaptation in brain lesion segmentation with adversarial networks
• Learning Compositional Visual Concepts with Mutual Consistency

Cited by in corpus (13)

• Generative Adversarial Network in Medical Imaging: A Review
• Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation
• Visual enhancement of Cone-beam CT by use of CycleGAN
• Joint Transmission Map Estimation and Dehazing using Deep Networks
• Simultaneous synthesis of FLAIR and segmentation of white matter hypointensities from T1 MRIs
• Unpaired Brain MR-to-CT Synthesis using a Structure-Constrained CycleGAN
• Constrained CycleGAN for Effective Generation of Ultrasound Sector Images of Improved Spatial Resolution
• Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation
• Image to Image Translation : Generating maps from satellite images
• C4Synth: Cross-Caption Cycle-Consistent Text-to-Image Synthesis
• Generator Versus Segmentor: Pseudo-healthy Synthesis
• Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks
• Generative Adversarial Networks for Video-to-Video Domain Adaptation