On instabilities of deep learning in image reconstruction - Does AI come at a cost?

Abstract

Deep learning, due to its unprecedented success in tasks such as image classification, has emerged as a new tool in image reconstruction with potential to change the field. In this paper we demonstrate a crucial phenomenon: deep learning typically yields unstablemethods for image reconstruction. The instabilities usually occur in several forms: (1) tiny, almost undetectable perturbations, both in the image and sampling domain, may result in severe artefacts in the reconstruction, (2) a small structural change, for example a tumour, may not be captured in the reconstructed image and (3) (a counterintuitive type of instability) more samples may yield poorer performance. Our new stability test with algorithms and easy to use software detects the instability phenomena. The test is aimed at researchers to test their networks for instabilities and for government agencies, such as the Food and Drug Administration (FDA), to secure safe use of deep learning methods.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Intriguing properties of neural networks
• Deep Convolutional Neural Network for Inverse Problems in Imaging
• Image reconstruction by domain transform manifold learning
• A Review of Convolutional Neural Networks for Inverse Problems in Imaging
• Solving ill-posed inverse problems using iterative deep neural networks
• Compressive Fluorescence Microscopy for Biological and Hyperspectral Imaging
• CNN-Based Projected Gradient Descent for Consistent Image Reconstruction
• MR image reconstruction using deep density priors
• Neural Proximal Gradient Descent for Compressive Imaging
• One Network to Solve Them All --- Solving Linear Inverse Problems using Deep Projection Models
• A multilevel based reweighting algorithm with joint regularizers for sparse recovery
• Shearlet-based compressed sensing for fast 3D cardiac MR imaging using iterative reweighting

Cited by in corpus (85)

• FISTA-Net: Learning A Fast Iterative Shrinkage Thresholding Network for Inverse Problems in Imaging
• Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction
• Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge
• Statistical Postprocessing for Weather Forecasts -- Review, Challenges and Avenues in a Big Data World
• Computed Tomography Reconstruction Using Deep Image Prior and Learned Reconstruction Methods
• Solving Inverse Problems With Deep Neural Networks -- Robustness Included?
• Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problem
• Solving Electrical Impedance Tomography with Deep Learning
• Do Deep Neural Networks Contribute to Multivariate Time Series Anomaly Detection?
• Measuring Robustness in Deep Learning Based Compressive Sensing
• Robust Compressed Sensing MRI with Deep Generative Priors
• Biomedical Image Reconstruction: From the Foundations to Deep Neural Networks
• Physics-based Reconstruction Methods for Magnetic Resonance Imaging
• An End-To-End-Trainable Iterative Network Architecture for Accelerated Radial Multi-Coil 2D Cine MR Image Reconstruction
• Multi-modal Deep Guided Filtering for Comprehensible Medical Image Processing
• Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant
• Ultra Low-Parameter Denoising: Trainable Bilateral Filter Layers in Computed Tomography
• Limitations of Deep Learning for Inverse Problems on Digital Hardware
• Multi-Resolution Data Fusion for Super Resolution Imaging
• Stable Deep MRI Reconstruction using Generative Priors
• Data driven regularization by projection
• Unsupervised Deep Basis Pursuit: Learning inverse problems without ground-truth data
• An Educated Warm Start For Deep Image Prior-Based Micro CT Reconstruction
• Trainable Joint Bilateral Filters for Enhanced Prediction Stability in Low-dose CT
• Score-Based Generative Models for PET Image Reconstruction
• Utilizing the Wavelet Transform's Structure in Compressed Sensing
• On Real-time Image Reconstruction with Neural Networks for MRI-guided Radiotherapy
• Quantitative Comparison of Planar Coded Aperture Imaging Reconstruction Methods
• What do AI algorithms actually learn? - On false structures in deep learning
• Improving Robustness of Deep-Learning-Based Image Reconstruction
• Unsupervised Knowledge-Transfer for Learned Image Reconstruction
• Physics-assisted Generative Adversarial Network for X-Ray Tomography
• Optimization methods for MR image reconstruction (long version)
• Data and Physics driven Deep Learning Models for Fast MRI Reconstruction: Fundamentals and Methodologies
• Learned, uncertainty-driven adaptive acquisition for photon-efficient scanning microscopy
• Deep Learning for Biomedical Image Reconstruction: A Survey
• Total Deep Variation: A Stable Regularizer for Inverse Problems
• The gap between theory and practice in function approximation with deep neural networks
• A Comparative Study of Deep Learning and Iterative Algorithms for Joint Channel Estimation and Signal Detection in OFDM Systems
• Spectroscopic data de-noising via training-set-free deep learning method
• Interpolated Joint Space Adversarial Training for Robust and Generalizable Defenses
• The Feasibility and Inevitability of Stealth Attacks
• Interval Neural Networks: Uncertainty Scores
• NESTANets: Stable, accurate and efficient neural networks for analysis-sparse inverse problems
• Deep Learning Techniques for Inverse Problems in Imaging
• A Tutorial on MRI Reconstruction: From Modern Methods to Clinical Implications
• Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie
• Compressed Sensing MRI Reconstruction Regularized by VAEs with Structured Image Covariance
• Advances in Photoacoustic Imaging Reconstruction and Quantitative Analysis for Biomedical Applications
• Trusted Artificial Intelligence: Towards Certification of Machine Learning Applications
• Fiducial marker recovery and detection from severely truncated data in navigation assisted spine surgery
• Data Consistent Artifact Reduction for Limited Angle Tomography with Deep Learning Prior
• Convex Regularization Behind Neural Reconstruction
• A toolkit of dilemmas: Beyond debiasing and fairness formulas for responsible AI/ML
• Validation and Generalizability of Self-Supervised Image Reconstruction Methods for Undersampled MRI
• Generative Adversarial Networks (GAN) Powered Fast Magnetic Resonance Imaging -- Mini Review, Comparison and Perspectives
• Self-supervised feature learning for cardiac Cine MR image reconstruction
• On Instabilities of Conventional Multi-Coil MRI Reconstruction to Small Adverserial Perturbations
• Localized adversarial artifacts for compressed sensing MRI
• SISMIK for brain MRI: Deep-learning-based motion estimation and model-based motion correction in k-space
• Systematic Review and Meta-analysis of AI-driven MRI Motion Artifact Detection and Correction
• Second Harmonic Imaging Enhanced by Deep Learning Decipher
• Multi-Channel Convolutional Analysis Operator Learning for Dual-Energy CT Reconstruction
• Enhancing quality and speed in database-free neural network reconstructions of undersampled MRI with SCAMPI
• Unsupervised Learning From Incomplete Measurements for Inverse Problems
• Explaining medical AI performance disparities across sites with confounder Shapley value analysis
• LAMA: Stable Dual-Domain Deep Reconstruction For Sparse-View CT
• Quantifying Model Uncertainty in Inverse Problems via Bayesian Deep Gradient Descent
• LAMA-Net: A Convergent Network Architecture for Dual-Domain Reconstruction
• Adaptive Local Neighborhood-based Neural Networks for MR Image Reconstruction from Undersampled Data
• On Ambiguity in Linear Inverse Problems: Entrywise Bounds on Nearly Data-Consistent Solutions and Entrywise Condition Numbers
• Data-Driven Theory-guided Learning of Partial Differential Equations using SimultaNeous Basis Function Approximation and Parameter Estimation (SNAPE)
• On the randomised stability constant for inverse problems
• Interpretation of Plug-and-Play (PnP) algorithms from a different angle
• Three-dimensional visualization of X-ray micro-CT with large-scale datasets: Efficiency and accuracy for real-time interaction
• Deep learning for 2D passive source detection in presence of complex cargo
• Instabilities in Plug-and-Play (PnP) algorithms from a learned denoiser
• Exploring Biologically Inspired Mechanisms of Adversarial Robustness
• Cine cardiac MRI reconstruction using a convolutional recurrent network with refinement
• Stabilizing Deep Tomographic Reconstruction
• Learning the geometry of wave-based imaging
• Structurally Adaptive Multi-Derivative Regularization for Image Recovery from Sparse Fourier Samples
• On $L^{\infty }$ stability for wave propagation and for linear inverse problems
• SUREMap: Predicting Uncertainty in CNN-based Image Reconstruction Using Stein's Unbiased Risk Estimate
• AAPM DL-Sparse-View CT Challenge Submission Report: Designing an Iterative Network for Fanbeam-CT with Unknown Geometry