Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN

Abstract

Generative Adversarial Networks (GAN) have many potential medical imaging applications, including data augmentation, domain adaptation, and model explanation. Due to the limited memory of Graphical Processing Units (GPUs), most current 3D GAN models are trained on low-resolution medical images, these models either cannot scale to high-resolution or are prone to patchy artifacts. In this work, we propose a novel end-to-end GAN architecture that can generate high-resolution 3D images. We achieve this goal by using different configurations between training and inference. During training, we adopt a hierarchical structure that simultaneously generates a low-resolution version of the image and a randomly selected sub-volume of the high-resolution image. The hierarchical design has two advantages: First, the memory demand for training on high-resolution images is amortized among sub-volumes. Furthermore, anchoring the high-resolution sub-volumes to a single low-resolution image ensures anatomical consistency between sub-volumes. During inference, our model can directly generate full high-resolution images. We also incorporate an encoder with a similar hierarchical structure into the model to extract features from the images. Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation. We also demonstrate clinical applications of the proposed model in data augmentation and clinical-relevant feature extraction.

arXiv commentPaper accepted to IEEE Journal of Biomedical and Health Informatics, code available at https://github.com/batmanlab/HA-GAN

References in corpus (9)

• Auto-Encoding Variational Bayes
• GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
• Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss
• MedGAN: Medical Image Translation using GANs
• 3D Convolutional Encoder-Decoder Network for Low-Dose CT via Transfer Learning from a 2D Trained Network
• Med3D: Transfer Learning for 3D Medical Image Analysis
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Artificial Intelligence in Glioma Imaging: Challenges and Advances
• Synthesizing Filamentary Structured Images with GANs

Cited by in corpus (7)

• Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges
• Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis
• WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesis
• 3D Brain and Heart Volume Generative Models: A Survey
• HAGAN: Hybrid Augmented Generative Adversarial Network for Medical Image Synthesis
• Devil is in Details: Locality-Aware 3D Abdominal CT Volume Generation for Self-Supervised Organ Segmentation
• Fréchet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets