Super-Resolution with Deep Convolutional Sufficient Statistics

Abstract

Inverse problems in image and audio, and super-resolution in particular, can be seen as high-dimensional structured prediction problems, where the goal is to characterize the conditional distribution of a high-resolution output given its low-resolution corrupted observation. When the scaling ratio is small, point estimates achieve impressive performance, but soon they suffer from the regression-to-the-mean problem, result of their inability to capture the multi-modality of this conditional distribution. Modeling high-dimensional image and audio distributions is a hard task, requiring both the ability to model complex geometrical structures and textured regions. In this paper, we propose to use as conditional model a Gibbs distribution, where its sufficient statistics are given by deep convolutional neural networks. The features computed by the network are stable to local deformation, and have reduced variance when the input is a stationary texture. These properties imply that the resulting sufficient statistics minimize the uncertainty of the target signals given the degraded observations, while being highly informative. The filters of the CNN are initialized by multiscale complex wavelets, and then we propose an algorithm to fine-tune them by estimating the gradient of the conditional log-likelihood, which bears some similarities with Generative Adversarial Networks. We evaluate experimentally the proposed approach in the image super-resolution task, but the approach is general and could be used in other challenging ill-posed problems such as audio bandwidth extension.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• DRAW: A Recurrent Neural Network For Image Generation
• Task-Driven Dictionary Learning
• Texture Synthesis Using Convolutional Neural Networks
• Deep Generative Stochastic Networks Trainable by Backprop
• Discriminative Regularization for Generative Models
• Audio Texture Synthesis with Scattering Moments
• The local low-dimensionality of natural images

Cited by in corpus (21)

• Density estimation using Real NVP
• Recurrent Generative Adversarial Networks for Proximal Learning and Automated Compressive Image Recovery
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• A Kinematic Bottleneck Approach For Pose Regression of Flexible Surgical Instruments directly from Images
• An inner-loop free solution to inverse problems using deep neural networks
• Diverse Image Synthesis from Semantic Layouts via Conditional IMLE
• InverseNet: Solving Inverse Problems with Splitting Networks
• Statistical learning of geometric characteristics of wireless networks
• Degrees of Freedom Analysis of Unrolled Neural Networks
• Unpaired Photo-to-Caricature Translation on Faces in the Wild
• WarpGAN: Automatic Caricature Generation
• Attaining Real-Time Super-Resolution for Microscopic Images Using GAN
• Photoacoustic Microscopy with Sparse Data Enabled by Convolutional Neural Networks for Fast Imaging
• Generative Adversarial Classifier for Handwriting Characters Super-Resolution
• Optimizing Generative Adversarial Networks for Image Super Resolution via Latent Space Regularization
• Generative Collaborative Networks for Single Image Super-Resolution
• Super-Resolution via Conditional Implicit Maximum Likelihood Estimation
• Denoising of 3-D Magnetic Resonance Images Using a Residual Encoder-Decoder Wasserstein Generative Adversarial Network
• Convolutional Neural Networks Analyzed via Inverse Problem Theory and Sparse Representations
• Analyzing Perception-Distortion Tradeoff using Enhanced Perceptual Super-resolution Network
• Instance Map based Image Synthesis with a Denoising Generative Adversarial Network