Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators

Abstract

Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration. A major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images. A surprising experiment that highlights this architectural bias towards natural images is that one can remove noise and corruptions from a natural image without using any training data, by simply fitting (via gradient descent) a randomly initialized, over-parameterized convolutional generator to the corrupted image. While this over-parameterized network can fit the corrupted image perfectly, surprisingly after a few iterations of gradient descent it generates an almost uncorrupted image. This intriguing phenomenon enables state-of-the-art CNN-based denoising and regularization of other inverse problems. In this paper, we attribute this effect to a particular architectural choice of convolutional networks, namely convolutions with fixed interpolating filters. We then formally characterize the dynamics of fitting a two-layer convolutional generator to a noisy signal and prove that early-stopped gradient descent denoises/regularizes. Our proof relies on showing that convolutional generators fit the structured part of an image significantly faster than the corrupted portion.

final ICRL version; simplifications in the proof

References in corpus (18)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Lossy Image Compression with Compressive Autoencoders
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Compressed Sensing with Deep Image Prior and Learned Regularization
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Recovery Guarantees for One-hidden-layer Neural Networks
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Algorithmic Guarantees for Inverse Imaging with Untrained Network Priors
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian
• Robust Compressive Phase Retrieval via Deep Generative Priors
• A Provably Convergent Scheme for Compressive Sensing under Random Generative Priors
• Phase Retrieval Under a Generative Prior
• Regularizing linear inverse problems with convolutional neural networks
• Stochastic Gradient Descent Learns State Equations with Nonlinear Activations

Cited by in corpus (9)

• Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximation
• Regularization Theory of the Analytic Deep Prior Approach
• Self-Validation: Early Stopping for Single-Instance Deep Generative Priors
• Deep Learning Techniques for Inverse Problems in Imaging
• Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterization
• Early Stopping in Deep Networks: Double Descent and How to Eliminate it
• Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
• Rank Overspecified Robust Matrix Recovery: Subgradient Method and Exact Recovery
• Untrained Graph Neural Networks for Denoising