Deep Image Prior

Abstract

Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, super-resolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs. Apart from its diverse applications, our approach highlights the inductive bias captured by standard generator network architectures. It also bridges the gap between two very popular families of image restoration methods: learning-based methods using deep convolutional networks and learning-free methods based on handcrafted image priors such as self-similarity. Code and supplementary material are available at https://dmitryulyanov.github.io/deep_image_prior .

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Understanding deep learning requires rethinking generalization
• Compressed Sensing with Deep Image Prior and Learned Regularization
• The Robust Manifold Defense: Adversarial Training using Generative Models
• Shift-Invariance Sparse Coding for Audio Classification
• Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
• Phase retrieval for Fourier Ptychography under varying amount of measurements
• Latent Convolutional Models

Cited by in corpus (148)

• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery
• DeepISP: Towards Learning an End-to-End Image Processing Pipeline
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
• Compressed Sensing with Deep Image Prior and Learned Regularization
• Image Restoration for Remote Sensing: Overview and Toolbox
• Noise2Inverse: Self-supervised deep convolutional denoising for tomography
• Mapping Degeneration Meets Label Evolution: Learning Infrared Small Target Detection with Single Point Supervision
• SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting
• The Robust Manifold Defense: Adversarial Training using Generative Models
• Image Compressed Sensing Using Non-local Neural Network
• To understand deep learning we need to understand kernel learning
• Instance Segmentation in the Dark
• Artificial Intelligence-Based Image Enhancement in PET Imaging: Noise Reduction and Resolution Enhancement
• Image Synthesis with a Single (Robust) Classifier
• Towards Automatic Concept-based Explanations
• Graph Unrolling Networks: Interpretable Neural Networks for Graph Signal Denoising
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Bridging the Gap Between Computational Photography and Visual Recognition
• Unsupervised Learning with Stein's Unbiased Risk Estimator
• Image Processing and Machine Learning for Hyperspectral Unmixing: An Overview and the HySUPP Python Package
• BCR-Net: a neural network based on the nonstandard wavelet form
• Direct PET Image Reconstruction Incorporating Deep Image Prior and a Forward Projection Model
• Efficient Continual Learning in Neural Networks with Embedding Regularization
• CARAFE: Content-Aware ReAssembly of FEatures
• Generative Models for Low-Rank Video Representation and Reconstruction
• Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images
• List-Mode PET Image Reconstruction Using Deep Image Prior
• Empty Cities: a Dynamic-Object-Invariant Space for Visual SLAM
• DeepRED: Deep Image Prior Powered by RED
• Probabilistic Autoencoder
• De-homogenization using Convolutional Neural Networks
• S2S-WTV: Seismic Data Noise Attenuation Using Weighted Total Variation Regularized Self-Supervised Learning
• Blind Image Deblurring with Unknown Kernel Size and Substantial Noise
• Graph Signal Restoration Using Nested Deep Algorithm Unrolling
• Compressible Latent-Space Invertible Networks for Generative Model-Constrained Image Reconstruction
• A multiscale neural network based on hierarchical matrices
• Deep Hyperspectral Prior: Denoising, Inpainting, Super-Resolution
• Fully 3D Implementation of the End-to-end Deep Image Prior-based PET Image Reconstruction Using Block Iterative Algorithm
• Inpainting Galactic Foreground Intensity and Polarization maps using Convolutional Neural Network
• Structural-analogy from a Single Image Pair
• Uncertainty Quantification with Generative Models
• Image Inpainting for Irregular Holes Using Partial Convolutions
• Memorization in Overparameterized Autoencoders
• When AWGN-based Denoiser Meets Real Noises
• An Unsupervised Approach to Solving Inverse Problems using Generative Adversarial Networks
• Image Deconvolution with Deep Image and Kernel Priors
• The Effectiveness of Instance Normalization: a Strong Baseline for Single Image Dehazing
• Imaging Conductivity from Current Density Magnitude using Neural Networks
• SINR: Deconvolving Circular SAS Images Using Implicit Neural Representations
• Faster Uncertainty Quantification for Inverse Problems with Conditional Normalizing Flows
• Integrating Deep Neural Networks with Full-waveform Inversion: Reparametrization, Regularization, and Uncertainty Quantification
• Self-Supervised Pre-Training for Deep Image Prior-Based Robust PET Image Denoising
• Removing grid structure in angle-resolved photoemission spectra via deep learning method
• Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser
• Exploring the Capability of Text-to-Image Diffusion Models with Structural Edge Guidance for Multi-Spectral Satellite Image Inpainting
• Modified Kernel MLAA Using Autoencoder for PET-enabled Dual-Energy CT
• Deep-Energy: Unsupervised Training of Deep Neural Networks
• Data and Physics driven Deep Learning Models for Fast MRI Reconstruction: Fundamentals and Methodologies
• Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample
• Semi-parametric Image Inpainting
• Non-Learning based Deep Parallel MRI Reconstruction (NLDpMRI)
• Neural Knitworks: Patched Neural Implicit Representation Networks
• Rank-One Network: An Effective Framework for Image Restoration
• Spectroscopic data de-noising via training-set-free deep learning method
• Image Processing Using Multi-Code GAN Prior
• Multi-Parameter Molecular MRI Quantification using Physics-Informed Self-Supervised Learning
• The Andromeda Gamma-Ray Excess: Background Systematics of the Millisecond Pulsars and Dark Matter Interpretations
• Scan-specific Self-supervised Bayesian Deep Non-linear Inversion for Undersampled MRI Reconstruction
• Occupancy Map Prediction Using Generative and Fully Convolutional Networks for Vehicle Navigation
• Extended Depth-of-Field Lensless Imaging using an Optimized Radial Mask
• Information Field Theory and Artificial Intelligence
• DeepSense: Enabling Carrier Sense in Low-Power Wide Area Networks Using Deep Learning
• Solving Inverse Problems in Steady-State Navier-Stokes Equations using Deep Neural Networks
• Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents
• A Tutorial on MRI Reconstruction: From Modern Methods to Clinical Implications
• An Underparametrized Deep Decoder Architecture for Graph Signals
• GAN2GAN: Generative Noise Learning for Blind Denoising with Single Noisy Images
• Déjà Vu: an empirical evaluation of the memorization properties of ConvNets
• Neural Architecture Search for Deep Image Prior
• Learning Invariant Representation for Unsupervised Image Restoration
• Logarithmic Pruning is All You Need
• End-to-end metasurface design for temperature imaging via broadband Planck-radiation regression
• Noise-robust latent vector reconstruction in ptychography using deep generative models
• Deformable image registration with deep network priors: a study on longitudinal PET images
• Ship in Sight: Diffusion Models for Ship-Image Super Resolution
• Neural topology optimization: the good, the bad, and the ugly
• Spacetime as the optimal generative network of quantum states: a roadmap to QM=GR?
• Invertible residual networks in the context of regularization theory for linear inverse problems
• Automatic Tuning of Denoising Algorithms Parameters Without Ground Truth
• Neural Blind Deconvolution Using Deep Priors
• Deep Learning Empowered Sub-Diffraction Terahertz Backpropagation Single-Pixel Imaging
• Model-free Optical Processors using In Situ Reinforcement Learning with Proximal Policy Optimization
• Perceptually Motivated Method for Image Inpainting Comparison
• A Test-Time Learning Approach to Reparameterize the Geophysical Inverse Problem with a Convolutional Neural Network
• Image registration and super resolution from first principles
• Direct Evolutionary Optimization of Variational Autoencoders With Binary Latents
• Imaging based on Compton scattering: model-uncertainty and data-driven reconstruction methods
• HDR Denoising and Deblurring by Learning Spatio-temporal Distortion Models
• Random mesh projectors for inverse problems
• Neural Photometry-guided Visual Attribute Transfer
• DDIPNet and DDIPNet+: Discriminant Deep Image Prior Networks for Remote Sensing Image Classification
• Learning Personalized Representation for Inverse Problems in Medical Imaging Using Deep Neural Network
• Task-Generic Hierarchical Human Motion Prior using VAEs
• Robust Equivariant Imaging: a fully unsupervised framework for learning to image from noisy and partial measurements
• NeRV: Neural Representations for Videos
• On the Convergence of Learning-based Iterative Methods for Nonconvex Inverse Problems
• MC-LSTM: Mass-Conserving LSTM
• Self-supervised physics-informed generative networks for phase retrieval from a single X-ray hologram
• Collaborative Sampling in Generative Adversarial Networks
• Towards Controllable and Photorealistic Region-wise Image Manipulation
• Exemplar-based synthesis of geology using kernel discrepancies and generative neural networks
• Rethinking Deep Image Prior for Denoising
• Disentangling Pose from Appearance in Monochrome Hand Images
• Distill Knowledge from NRSfM for Weakly Supervised 3D Pose Learning
• Cost Function Unrolling in Unsupervised Optical Flow
• Deep High-Resolution Network for Low Dose X-ray CT Denoising
• Bi-Skip: A Motion Deblurring Network Using Self-paced Learning
• A multiscale neural network based on hierarchical nested bases
• Deep Optimized Priors for 3D Shape Modeling and Reconstruction
• Janssen 2.0: Audio Inpainting in the Time-frequency Domain
• Exploiting network optimization stability for enhanced PET image denoising using deep image prior
• Limited-angle SPECT image reconstruction using deep image prior
• Motion Selective Prediction for Video Frame Synthesis
• Deep Likelihood Network for Image Restoration with Multiple Degradation Levels
• J-Net: Randomly weighted U-Net for audio source separation
• Unsupervised Adversarial Image Inpainting
• Deep Interactive Denoiser (DID) for X-Ray Computed Tomography
• A Methodology for Exploring Deep Convolutional Features in Relation to Hand-Crafted Features with an Application to Music Audio Modeling
• Correlated Read Noise Reduction in Infrared Arrays Using Deep Learning
• Non-Convex Compressed Sensing with Training Data
• JPEG Information Regularized Deep Image Prior for Denoising
• Deep Gradient Projection Networks for Pan-sharpening
• Representation, Analysis of Bayesian Refinement Approximation Network: A Survey
• Training Deep Normalizing Flow Models in Highly Incomplete Data Scenarios with Prior Regularization
• Gradient Kernel Regression
• Photo style transfer with consistency losses
• Quantitative mapping from conventional MRI using self-supervised physics-guided deep learning: applications to a large-scale, clinically heterogeneous dataset
• Robust Trust Region for Weakly Supervised Segmentation
• Neural Weight Step Video Compression
• Improved Image Generation via Sparse Modeling
• JR2net: A Joint Non-Linear Representation and Recovery Network for Compressive Spectral Imaging
• Class-Specific Blind Deconvolutional Phase Retrieval Under a Generative Prior
• Untrained Graph Neural Networks for Denoising
• On the Frequency Bias of Generative Models
• Meta Internal Learning