Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging

Abstract

Since the advent of deep convolutional neural networks (DNNs), computer vision has seen an extremely rapid progress that has led to huge advances in medical imaging. This article does not aim to cover all aspects of the field but focuses on a particular topic, image-to-image translation. Although the topic may not sound familiar, it turns out that many seemingly irrelevant applications can be understood as instances of image-to-image translation. Such applications include (1) noise reduction, (2) super-resolution, (3) image synthesis, and (4) reconstruction. The same underlying principles and algorithms work for various tasks. Our aim is to introduce some of the key ideas on this topic from a uniform point of view. We introduce core ideas and jargon that are specific to image processing by use of DNNs. Having an intuitive grasp of the core ideas of and a knowledge of technical terms would be of great help to the reader for understanding the existing and future applications. Most of the recent applications which build on image-to-image translation are based on one of two fundamental architectures, called pix2pix and CycleGAN, depending on whether the available training data are paired or unpaired. We provide computer codes which implement these two architectures with various enhancements. Our codes are available online with use of the very permissive MIT license. We provide a hands-on tutorial for training a model for denoising based on our codes. We hope that this article, together with the codes, will provide both an overview and the details of the key algorithms, and that it will serve as a basis for the development of new applications.

arXiv commentmany typos are fixed. to appear in Radiological Physics and Technology

References in corpus (10)

• A Survey on Deep Learning in Medical Image Analysis
• Image reconstruction by domain transform manifold learning
• Low Dose CT Image Denoising Using a Generative Adversarial Network with Wasserstein Distance and Perceptual Loss
• A guide to convolution arithmetic for deep learning
• Sharpness-aware Low dose CT denoising using conditional generative adversarial network
• Cycle Consistent Adversarial Denoising Network for Multiphase Coronary CT Angiography
• Learning Implicit Brain MRI Manifolds with Deep Learning
• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
• Generative Adversarial Networks for Image-to-Image Translation on Multi-Contrast MR Images - A Comparison of CycleGAN and UNIT
• Bayesian Image Quality Transfer with CNNs: Exploring Uncertainty in dMRI Super-Resolution