Image reconstruction by domain transform manifold learning

Abstract

Image reconstruction plays a critical role in the implementation of all contemporary imaging modalities across the physical and life sciences including optical, MRI, CT, PET, and radio astronomy. During an image acquisition, the sensor encodes an intermediate representation of an object in the sensor domain, which is subsequently reconstructed into an image by an inversion of the encoding function. Image reconstruction is challenging because analytic knowledge of the inverse transform may not exist a priori, especially in the presence of sensor non-idealities and noise. Thus, the standard reconstruction approach involves approximating the inverse function with multiple ad hoc stages in a signal processing chain whose composition depends on the details of each acquisition strategy, and often requires expert parameter tuning to optimize reconstruction performance. We present here a unified framework for image reconstruction, AUtomated TransfOrm by Manifold APproximation (AUTOMAP), which recasts image reconstruction as a data-driven, supervised learning task that allows a mapping between sensor and image domain to emerge from an appropriate corpus of training data. We implement AUTOMAP with a deep neural network and exhibit its flexibility in learning reconstruction transforms for a variety of MRI acquisition strategies, using the same network architecture and hyperparameters. We further demonstrate its efficiency in sparsely representing transforms along low-dimensional manifolds, resulting in superior immunity to noise and reconstruction artifacts compared with conventional handcrafted reconstruction methods. In addition to improving the reconstruction performance of existing acquisition methodologies, we anticipate accelerating the discovery of new acquisition strategies across modalities as the burden of reconstruction becomes lifted by AUTOMAP and learned-reconstruction approaches.

18 pages, 4 figures

References in corpus (5)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Deep Convolutional Neural Network for Inverse Problems in Imaging
• Winner-Take-All Autoencoders
• Sparse representations and convex optimization as tools for LOFAR radio interferometric imaging
• A general spline representation for nonparametric and semiparametric density estimates using diffeomorphisms

Cited by in corpus (203)

• An overview of deep learning in medical imaging focusing on MRI
• TransMorph: Transformer for unsupervised medical image registration
• On instabilities of deep learning in image reconstruction - Does AI come at a cost?
• fastMRI: An Open Dataset and Benchmarks for Accelerated MRI
• Deep Learning in Medical Image Registration: A Survey
• Can Deep Learning Outperform Modern Commercial CT Image Reconstruction Methods?
• Self-Supervised Learning of Physics-Guided Reconstruction Neural Networks without Fully-Sampled Reference Data
• DeepPET: A deep encoder-decoder network for directly solving the PET reconstruction inverse problem
• Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction
• Image Reconstruction: From Sparsity to Data-adaptive Methods and Machine Learning
• Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge
• Plug and play methods for magnetic resonance imaging (long version)
• Adaptive Diffusion Priors for Accelerated MRI Reconstruction
• InversionNet: A Real-Time and Accurate Full Waveform Inversion with CNNs and continuous CRFs
• A Review on Deep Learning in Medical Image Reconstruction
• Computed Tomography Reconstruction Using Deep Image Prior and Learned Reconstruction Methods
• MR image reconstruction using deep density priors
• Deep Learning in Photoacoustic Tomography: Current approaches and future directions
• Solving Inverse Problems With Deep Neural Networks -- Robustness Included?
• Learning with Known Operators reduces Maximum Training Error Bounds
• Method for motion artifact reduction using a convolutional neural network for dynamic contrast enhanced MRI of the liver
• Convolutional Sparse Coding for Compressed Sensing CT Reconstruction
• Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problem
• Optical coherent dot-product chip for sophisticated deep learning regression
• Real-time coherent diffraction inversion using deep generative networks
• J-MoDL: Joint Model-Based Deep Learning for Optimized Sampling and Reconstruction
• Spatio-Temporal Deep Learning-Based Undersampling Artefact Reduction for 2D Radial Cine MRI with Limited Data
• Predicting Lung Nodule Malignancies by Combining Deep Convolutional Neural Network and Handcrafted Features
• Deep learning in biomedical optics
• Deep variational network for rapid 4D flow MRI reconstruction
• MoCoNet: Motion Correction in 3D MPRAGE images using a Convolutional Neural Network approach
• PYRO-NN: Python Reconstruction Operators in Neural Networks
• Wasserstein GANs for MR Imaging: from Paired to Unpaired Training
• Performance characterization of a novel deep learning-based MR image reconstruction pipeline
• MRzero -- Fully automated discovery of MRI sequences using supervised learning
• Opening a new window on MR-based Electrical Properties Tomography with deep learning
• Robust Phase Unwrapping via Deep Image Prior for Quantitative Phase Imaging
• An End-to-End AI-Based Framework for Automated Discovery of CEST/MT MR Fingerprinting Acquisition Protocols and Quantitative Deep Reconstruction (AutoCEST)
• Real-time 3D Nanoscale Coherent Imaging via Physics-aware Deep Learning
• Multi-Modal MRI Reconstruction Assisted with Spatial Alignment Network
• Training a Neural Network for Gibbs and Noise Removal in Diffusion MRI
• xQSM: Quantitative Susceptibility Mapping with Octave Convolutional and Noise Regularized Neural Networks
• Direct PET Image Reconstruction Incorporating Deep Image Prior and a Forward Projection Model
• Dual-Domain Reconstruction Networks with V-Net and K-Net for fast MRI
• Separation of water and fat signal in whole-body gradient echo scans using convolutional neural networks
• Deep Efficient End-to-end Reconstruction (DEER) Network for Few-view Breast CT Image Reconstruction
• Physics-/Model-Based and Data-Driven Methods for Low-Dose Computed Tomography: A survey
• LORAKI: Autocalibrated Recurrent Neural Networks for Autoregressive MRI Reconstruction in k-Space
• Biomedical Image Reconstruction: From the Foundations to Deep Neural Networks
• A Helmholtz equation solver using unsupervised learning: Application to transcranial ultrasound
• List-Mode PET Image Reconstruction Using Deep Image Prior
• Active MR k-space Sampling with Reinforcement Learning
• Neural Networks-based Regularization for Large-Scale Medical Image Reconstruction
• Deep MRI Reconstruction: Unrolled Optimization Algorithms Meet Neural Networks
• Scan Specific Artifact Reduction in K-space (SPARK) Neural Networks Synergize with Physics-based Reconstruction to Accelerate MRI
• Artifact- and content-specific quality assessment for MRI with image rulers
• DUG-RECON: A Framework for Direct Image Reconstruction using Convolutional Generative Networks
• Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future
• Training Variational Networks with Multi-Domain Simulations: Speed-of-Sound Image Reconstruction
• An Unsupervised Deep Learning Method for Multi-coil Cine MRI
• Report on the AAPM deep-learning sparse-view CT (DL-sparse-view CT) Grand Challenge
• Systematic Review on Learning-based Spectral CT
• Noise2Context: Context-assisted Learning 3D Thin-layer Low Dose CT Without Clean Data
• OCMR (v1.0)--Open-Access Multi-Coil k-Space Dataset for Cardiovascular Magnetic Resonance Imaging
• High signal-to-noise ratio reconstruction of low bit-depth optical coherence tomography using deep learning
• Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
• Accelerating Quantitative Susceptibility Mapping using Compressed Sensing and Deep Neural Network
• An Unsupervised Reconstruction Method For Low-Dose CT Using Deep Generative Regularization Prior
• Complex Network for Complex Problems: A comparative study of CNN and Complex-valued CNN
• Limitations of Deep Learning for Inverse Problems on Digital Hardware
• Neural Architecture Search for Compressed Sensing Magnetic Resonance Image Reconstruction
• Molecular MRI-Based Monitoring of Cancer Immunotherapy Treatment Response
• X2CT-GAN: Reconstructing CT from Biplanar X-Rays with Generative Adversarial Networks
• Bi-Linear Modeling of Data Manifolds for Dynamic-MRI Recovery
• Stable Deep MRI Reconstruction using Generative Priors
• Compressible Latent-Space Invertible Networks for Generative Model-Constrained Image Reconstruction
• dAUTOMAP: decomposing AUTOMAP to achieve scalability and enhance performance
• Fully 3D Implementation of the End-to-end Deep Image Prior-based PET Image Reconstruction Using Block Iterative Algorithm
• Joint Frequency and Image Space Learning for MRI Reconstruction and Analysis
• Data-Consistent Non-Cartesian Deep Subspace Learning for Efficient Dynamic MR Image Reconstruction
• Geometric Approaches to Increase the Expressivity of Deep Neural Networks for MR Reconstruction
• Utilizing the Wavelet Transform's Structure in Compressed Sensing
• CoIL: Coordinate-based Internal Learning for Imaging Inverse Problems
• $\Sigma$-net: Systematic Evaluation of Iterative Deep Neural Networks for Fast Parallel MR Image Reconstruction
• Deep, Deep Learning with BART
• k-Space Deep Learning for Accelerated MRI
• DeepcomplexMRI: Exploiting deep residual network for fast parallel MR imaging with complex convolution
• Quantitative Comparison of Planar Coded Aperture Imaging Reconstruction Methods
• Learned convex regularizers for inverse problems
• On Real-time Image Reconstruction with Neural Networks for MRI-guided Radiotherapy
• Improving Robustness of Deep-Learning-Based Image Reconstruction
• Physics-assisted Generative Adversarial Network for X-Ray Tomography
• What do AI algorithms actually learn? - On false structures in deep learning
• Assessment of Data Consistency through Cascades of Independently Recurrent Inference Machines for fast and robust accelerated MRI reconstruction
• Deep Learning for Biomedical Image Reconstruction: A Survey
• A deep cascade of ensemble of dual domain networks with gradient-based T1 assistance and perceptual refinement for fast MRI reconstruction
• PtychoDV: Vision Transformer-Based Deep Unrolling Network for Ptychographic Image Reconstruction
• End-to-end reconstruction meets data-driven regularization for inverse problems
• Analysis of Deep Complex-Valued Convolutional Neural Networks for MRI Reconstruction
• Non-Learning based Deep Parallel MRI Reconstruction (NLDpMRI)
• AutoSamp: Autoencoding k-space Sampling via Variational Information Maximization for 3D MRI
• Dual Network Architecture for Few-view CT -- Trained on ImageNet Data and Transferred for Medical Imaging
• Compressed Sensing with Signal Averaging for Improved Sensitivity and Motion Artifact Reduction in Fluorine-19 MRI
• Machine Learning for Improved Current Density Reconstruction from 2D Vector Magnetic Images
• Bayesian Image Reconstruction using Deep Generative Models
• TransCT: Dual-path Transformer for Low Dose Computed Tomography
• Reducing Uncertainty in Undersampled MRI Reconstruction with Active Acquisition
• Efficient Physics-Based Learned Reconstruction Methods for Real-Time 3D Near-Field MIMO Radar Imaging
• Universal Deep Beamformer for Variable Rate Ultrasound Imaging
• Geometry Constrained Weakly Supervised Object Localization
• Appearance Learning for Image-based Motion Estimation in Tomography
• Deep Learning Techniques for Inverse Problems in Imaging
• DIMENSION: Dynamic MR Imaging with Both K-space and Spatial Prior Knowledge Obtained via Multi-Supervised Network Training
• A Tutorial on MRI Reconstruction: From Modern Methods to Clinical Implications
• Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging
• Learning Geometry-Dependent and Physics-Based Inverse Image Reconstruction
• FastPET: Near Real-Time PET Reconstruction from Histo-Images Using a Neural Network
• Deep Learning Based Computed Tomography Whys and Wherefores
• Latent Signal Models: Learning Compact Representations of Signal Evolution for Improved Time-Resolved, Multi-contrast MRI
• Compressed Sensing MRI Reconstruction Regularized by VAEs with Structured Image Covariance
• A deep network for sinogram and CT image reconstruction
• Complex Fully Convolutional Neural Networks for MR Image Reconstruction
• KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in MRI workflow
• SANTIS: Sampling-Augmented Neural neTwork with Incoherent Structure for MR image reconstruction
• Task Transformer Network for Joint MRI Reconstruction and Super-Resolution
• Model-based Deep Medical Imaging: the roadmap of generalizing iterative reconstruction model using deep learning
• ReconU-Net: a direct PET image reconstruction using U-Net architecture with back projection-induced skip connection
• MRI Reconstruction with Regularized 3D Diffusion Model (R3DM)
• MRI Image Reconstruction via Learning Optimization Using Neural ODEs
• Localized adversarial artifacts for compressed sensing MRI
• Machine Friendly Machine Learning: Interpretation of Computed Tomography Without Image Reconstruction
• Deep learning brain conductivity mapping using a patch-based 3D U-net
• Model Learning: Primal Dual Networks for Fast MR imaging
• Optimal MRI Undersampling Patterns for Ultimate Benefit of Medical Vision Tasks
• Conditional WGANs with Adaptive Gradient Balancing for Sparse MRI Reconstruction
• Sinogram interpolation for sparse-view micro-CT with deep learning neural network
• A New Deep Learning Method for Image Deblurring in Optical Microscopic Systems
• Reconstruction of Cardiac Cine MRI Using Motion-Guided Deformable Alignment and Multi-Resolution Fusion
• Deep-learning Image Reconstruction for Real-time Photoacoustic System
• Random mesh projectors for inverse problems
• AliasNet: Alias Artefact Suppression Network for Accelerated Phase-Encode MRI
• Efficient B-mode Ultrasound Image Reconstruction from Sub-sampled RF Data using Deep Learning
• MRI Pulse Sequence Integration for Deep-Learning Based Brain Metastasis Segmentation
• Deriving Neural Network Architectures using Precision Learning: Parallel-to-fan beam Conversion
• Rethinking Medical Image Reconstruction via Shape Prior, Going Deeper and Faster: Deep Joint Indirect Registration and Reconstruction
• Adaptive convolutional neural networks for k-space data interpolation in fast magnetic resonance imaging
• Deep Neural Network Assisted Iterative Reconstruction Method for Low Dose CT
• AI-Enabled Ultra-Low-Dose CT Reconstruction
• Transform and Bitstream Domain Image Classification
• CT Reconstruction with PDF: Parameter-Dependent Framework for Multiple Scanning Geometries and Dose Levels
• Robustness Investigation on Deep Learning CT Reconstruction for Real-Time Dose Optimization
• Analog-to-digital conversion revolutionized by deep learning
• Deep-ER: Deep Learning ECCENTRIC Reconstruction for fast high-resolution neurometabolic imaging
• Deep learning for fast MR imaging: a review for learning reconstruction from incomplete k-space data
• Quantifying Model Uncertainty in Inverse Problems via Bayesian Deep Gradient Descent
• Free-breathing Cardiovascular MRI Using a Plug-and-Play Method with Learned Denoiser
• Data Consistent CT Reconstruction from Insufficient Data with Learned Prior Images
• RDAnet: A Deep Learning Based Approach for Synthetic Aperture Radar Image Formation
• 3D U-NetR: Low Dose Computed Tomography Reconstruction via Deep Learning and 3 Dimensional Convolutions
• Magnetic Resonance Parameter Mapping using Self-supervised Deep Learning with Model Reinforcement
• From the Greene--Wu Convolution to Gradient Estimation over Riemannian Manifolds
• Towards Ultrafast MRI via Extreme k-Space Undersampling and Superresolution
• Real-Time Mapping of Tissue Properties for Magnetic Resonance Fingerprinting
• A Few-Shot Learning Approach for Accelerated MRI via Fusion of Data-Driven and Subject-Driven Priors
• Applications of Deep Learning for Ill-Posed Inverse Problems Within Optical Tomography
• Relaxometry Guided Quantitative Cardiac Magnetic Resonance Image Reconstruction
• MetaInv-Net: Meta Inversion Network for Sparse View CT Image Reconstruction
• Defending Water Treatment Networks: Exploiting Spatio-temporal Effects for Cyber Attack Detection
• Detection and Correction of Cardiac MR Motion Artefacts during Reconstruction from K-space
• IFR-Net: Iterative Feature Refinement Network for Compressed Sensing MRI
• On the randomised stability constant for inverse problems
• Learning image from projection: a full-automatic reconstruction (FAR) net for sparse-views computed tomography
• Artificial Intelligence-Guided PET Image Reconstruction and Multi-Tracer Imaging: Novel Methods, Challenges, And Opportunities
• Learning Priors in High-frequency Domain for Inverse Imaging Reconstruction
• Deep learning scheme for recovery of broadband microwave photonic receiving systems in transceivers without expert knowledge and system priors
• Improving Generalization of Sequence Encoder-Decoder Networks for Inverse Imaging of Cardiac Transmembrane Potential
• MANTIS: Model-Augmented Neural neTwork with Incoherent k-space Sampling for efficient MR T2 mapping
• Stabilizing Deep Tomographic Reconstruction
• DeepCEST: 9.4 T Chemical Exchange Saturation Transfer MRI contrast predicted from 3 T data - a proof of concept study
• Accelerating MR Imaging via Deep Chambolle-Pock Network
• Interpretation of Plug-and-Play (PnP) algorithms from a different angle
• Robust diffusion parametric mapping of motion-corrupted data with a three-dimensional convolutional neural network
• Seeing through a Black Box: Toward High-Quality Terahertz TomographicImaging via Multi-Scale Spatio-Spectral Image Fusion
• DONet: Dual-Octave Network for Fast MR Image Reconstruction
• Three-dimensional visualization of X-ray micro-CT with large-scale datasets: Efficiency and accuracy for real-time interaction
• SUPER Learning: A Supervised-Unsupervised Framework for Low-Dose CT Image Reconstruction
• A hierarchical approach to deep learning and its application to tomographic reconstruction
• Split Slice Training Augmentation and Hyperparameter Tuning of RAKI Networks for Simultaneous Multi-Slice Reconstruction
• CRDN: Cascaded Residual Dense Networks for Dynamic MR Imaging with Edge-enhanced Loss Constraint
• Deep Manifold Embedding for Hyperspectral Image Classification
• An Exploration of Parallel Imaging System for Very-low Field (50mT) MRI Scanner
• Unpaired Deep Learning for Accelerated MRI using Optimal Transport Driven CycleGAN
• Is good old GRAPPA dead?
• APIR-Net: Autocalibrated Parallel Imaging Reconstruction using a Neural Network
• MD-Recon-Net: A Parallel Dual-Domain Convolutional Neural Network for Compressed Sensing MRI
• Hardware-irrelevant parallel processing system
• Optimal Design of Experiments on Riemannian Manifolds
• Reconstructing unseen modalities and pathology with an efficient Recurrent Inference Machine
• A General Framework for Inverse Problem Solving using Self-Supervised Deep Learning: Validations in Ultrasound and Photoacoustic Image Reconstruction
• Joint Motion Correction and Super Resolution for Cardiac Segmentation via Latent Optimisation
• Do CNNs solve the CT inverse problem?
• UVTomo-GAN: An adversarial learning based approach for unknown view X-ray tomographic reconstruction
• An Adversarial Learning Based Approach for Unknown View Tomographic Reconstruction