Amortised MAP Inference for Image Super-resolution

Abstract

Image super-resolution (SR) is an underdetermined inverse problem, where a large number of plausible high-resolution images can explain the same downsampled image. Most current single image SR methods use empirical risk minimisation, often with a pixel-wise mean squared error (MSE) loss. However, the outputs from such methods tend to be blurry, over-smoothed and generally appear implausible. A more desirable approach would employ Maximum a Posteriori (MAP) inference, preferring solutions that always have a high probability under the image prior, and thus appear more plausible. Direct MAP estimation for SR is non-trivial, as it requires us to build a model for the image prior from samples. Furthermore, MAP inference is often performed via optimisation-based iterative algorithms which don't compare well with the efficiency of neural-network-based alternatives. Here we introduce new methods for amortised MAP inference whereby we calculate the MAP estimate directly using a convolutional neural network. We first introduce a novel neural network architecture that performs a projection to the affine subspace of valid SR solutions ensuring that the high resolution output of the network is always consistent with the low resolution input. We show that, using this architecture, the amortised MAP inference problem reduces to minimising the cross-entropy between two distributions, similar to training generative models. We propose three methods to solve this optimisation problem: (1) Generative Adversarial Networks (GAN) (2) denoiser-guided SR which backpropagates gradient-estimates from denoising to train the network, and (3) a baseline method using a maximum-likelihood-trained image prior. Our experiments show that the GAN based approach performs best on real image data. Lastly, we establish a connection between GANs and amortised variational inference as in e.g. variational autoencoders.

Cited by in corpus (76)

• Generative Adversarial Networks: An Overview
• ICLabel: An automated electroencephalographic independent component classifier, dataset, and website
• Diffusion Models in Vision: A Survey
• Reconstruction of three-dimensional porous media using generative adversarial neural networks
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• Plug and play methods for magnetic resonance imaging (long version)
• Good Semi-supervised Learning that Requires a Bad GAN
• Stabilizing Training of Generative Adversarial Networks through Regularization
• The Numerics of GANs
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Isolating Sources of Disentanglement in Variational Autoencoders
• Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates
• Image Augmentations for GAN Training
• From Variational to Deterministic Autoencoders
• Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step
• Differentiable Augmentation for Data-Efficient GAN Training
• Fashion-Gen: The Generative Fashion Dataset and Challenge
• Deep Learning for Image Super-resolution: A Survey
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Wasserstein GANs for MR Imaging: from Paired to Unpaired Training
• Do Deep Generative Models Know What They Don't Know?
• Gradient Estimators for Implicit Models
• Streetscape augmentation using generative adversarial networks: insights related to health and wellbeing
• Physics-Based Generative Adversarial Models for Image Restoration and Beyond
• ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis
• How Well Generative Adversarial Networks Learn Distributions
• Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
• Generative Modeling with Conditional Autoencoders: Building an Integrated Cell
• High-resolution Deep Convolutional Generative Adversarial Networks
• MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks
• Mode Collapse and Regularity of Optimal Transportation Maps
• Responsible Disclosure of Generative Models Using Scalable Fingerprinting
• Encoding Invariances in Deep Generative Models
• Wasserstein-2 Generative Networks
• A Single Video Super-Resolution GAN for Multiple Downsampling Operators based on Pseudo-Inverse Image Formation Models
• LCSCNet: Linear Compressing Based Skip-Connecting Network for Image Super-Resolution
• Semi-supervised Rare Disease Detection Using Generative Adversarial Network
• Finding Mixed Nash Equilibria of Generative Adversarial Networks
• Ranking CGANs: Subjective Control over Semantic Image Attributes
• Fast and Robust Cascade Model for Multiple Degradation Single Image Super-Resolution
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)
• Global Guarantees for Blind Demodulation with Generative Priors
• Variational Implicit Processes
• Non-saturating GAN training as divergence minimization
• Fast Face-swap Using Convolutional Neural Networks
• GridFace: Face Rectification via Learning Local Homography Transformations
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts
• Dual Contrastive Loss and Attention for GANs
• Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment
• Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference
• LUCSS: Language-based User-customized Colourization of Scene Sketches
• Random mesh projectors for inverse problems
• Learning Implicit Generative Models with Theoretical Guarantees
• Training Generative Adversarial Networks from Incomplete Observations using Factorised Discriminators
• Embodied Self-supervised Learning by Coordinated Sampling and Training
• Problems using deep generative models for probabilistic audio source separation
• Deep Decomposition Learning for Inverse Imaging Problems
• Self-supervised Fine-tuning for Correcting Super-Resolution Convolutional Neural Networks
• A Decoupled 3D Facial Shape Model by Adversarial Training
• Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis
• Generative Learning With Euler Particle Transport
• Nonasymptotic Guarantees for Spiked Matrix Recovery with Generative Priors
• Robust Density Estimation under Besov IPM Losses
• Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space
• Fighting Copycat Agents in Behavioral Cloning from Observation Histories
• MIM-Based GAN: Information Metric to Amplify Small Probability Events Importance in Generative Adversarial Networks
• Adaptive Divergence for Rapid Adversarial Optimization
• Deep Likelihood Network for Image Restoration with Multiple Degradation Levels
• Deep 3D Pan via adaptive "t-shaped" convolutions with global and local adaptive dilations
• Analysis of Discriminator in RKHS Function Space for Kullback-Leibler Divergence Estimation
• Machine Vision for Improved Human-Robot Cooperation in Adverse Underwater Conditions
• ReshapeGAN: Object Reshaping by Providing A Single Reference Image
• High Diversity Attribute Guided Face Generation with GANs
• Learning color space adaptation from synthetic to real images of cirrus clouds
• Local Stability and Performance of Simple Gradient Penalty mu-Wasserstein GAN