ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks

Abstract

The Super-Resolution Generative Adversarial Network (SRGAN) is a seminal work that is capable of generating realistic textures during single image super-resolution. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly study three key components of SRGAN - network architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN). In particular, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. Moreover, we borrow the idea from relativistic GAN to let the discriminator predict relative realness instead of the absolute value. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Benefiting from these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN and won the first place in the PIRM2018-SR Challenge. The code is available at https://github.com/xinntao/ESRGAN .

arXiv commentTo appear in ECCV 2018 workshop. Won Region 3 in the PIRM2018-SR Challenge. Code and models are at https://github.com/xinntao/ESRGAN

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Spectral Normalization for Generative Adversarial Networks
• The Perception-Distortion Tradeoff
• The relativistic discriminator: a key element missing from standard GAN
• Texture Synthesis Using Convolutional Neural Networks
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Image Super-Resolution Using Very Deep Residual Channel Attention Networks
• Residual Dense Network for Image Super-Resolution
• MemNet: A Persistent Memory Network for Image Restoration
• Unsupervised Image Super-Resolution using Cycle-in-Cycle Generative Adversarial Networks

Cited by in corpus (201)

• The Perception-Distortion Tradeoff
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Stochastic Super-Resolution for Downscaling Time-Evolving Atmospheric Fields with a Generative Adversarial Network
• CTCNet: A CNN-Transformer Cooperation Network for Face Image Super-Resolution
• Unfolding the Alternating Optimization for Blind Super Resolution
• HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment
• Structure-Preserving Image Super-Resolution
• Path-Restore: Learning Network Path Selection for Image Restoration
• Data-Free Adversarial Distillation
• StarGAN v2: Diverse Image Synthesis for Multiple Domains
• A Deep Journey into Super-resolution: A survey
• Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data
• Deformable Non-local Network for Video Super-Resolution
• SwinIR: Image Restoration Using Swin Transformer
• MFRNet: A New CNN Architecture for Post-Processing and In-loop Filtering
• Deep Learning for Image Super-resolution: A Survey
• SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects
• Learning Enriched Features for Real Image Restoration and Enhancement
• Attention in Attention Network for Image Super-Resolution
• EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
• NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling
• Residual Dense Network for Image Restoration
• A Latent Encoder Coupled Generative Adversarial Network (LE-GAN) for Efficient Hyperspectral Image Super-resolution
• Multi-Stage Progressive Image Restoration
• Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives
• Toward Real-World Super-Resolution via Adaptive Downsampling Models
• Feedback Network for Image Super-Resolution
• Learning Texture Transformer Network for Image Super-Resolution
• Efficient Image Super-Resolution Using Pixel Attention
• TWIST-GAN: Towards Wavelet Transform and Transferred GAN for Spatio-Temporal Single Image Super Resolution
• Image Fine-grained Inpainting
• Structure-Preserving Super Resolution with Gradient Guidance
• Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments
• Unsupervised Degradation Learning for Single Image Super-Resolution
• Cephalogram Synthesis and Landmark Detection in Dental Cone-Beam CT Systems
• Gated Multiple Feedback Network for Image Super-Resolution
• PyNET-CA: Enhanced PyNET with Channel Attention for End-to-End Mobile Image Signal Processing
• Towards Real-World Blind Face Restoration with Generative Facial Prior
• Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives
• Blind Super-Resolution With Iterative Kernel Correction
• NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results
• Real-World Single Image Super-Resolution: A Brief Review
• Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Human-Like Sketches
• Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Clouds
• AIM 2020 Challenge on Real Image Super-Resolution: Methods and Results
• AIM 2019 Challenge on Constrained Super-Resolution: Methods and Results
• Uncovering the Over-smoothing Challenge in Image Super-Resolution: Entropy-based Quantification and Contrastive Optimization
• Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation
• A Single Video Super-Resolution GAN for Multiple Downsampling Operators based on Pseudo-Inverse Image Formation Models
• Interpreting Super-Resolution Networks with Local Attribution Maps
• Toward DNN of LUTs: Learning Efficient Image Restoration with Multiple Look-Up Tables
• CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement
• A Specific Task-oriented Semantic Image Communication System for substation patrol inspection
• Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark Estimation
• The Six Fronts of the Generative Adversarial Networks
• SRGAN: Training Dataset Matters
• SRDGAN: learning the noise prior for Super Resolution with Dual Generative Adversarial Networks
• Two-stage Deep Denoising with Self-guided Noise Attention for Multimodal Medical Images
• Distributed Learning and Inference with Compressed Images
• Progressive Semantic-Aware Style Transformation for Blind Face Restoration
• SynFi: Automatic Synthetic Fingerprint Generation
• Blind Face Restoration via Deep Multi-scale Component Dictionaries
• Infrared Image Super-Resolution via Heterogeneous Convolutional WGAN
• SegDiff: Image Segmentation with Diffusion Probabilistic Models
• Deep Model-Based Super-Resolution with Non-uniform Blur
• Conditional Generative Adversarial Networks for Channel Estimation in RIS-Assisted ISAC Systems
• Content and Colour Distillation for Learning Image Translations with the Spatial Profile Loss
• Deblurring by Realistic Blurring
• Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization
• Bayesian Image Reconstruction using Deep Generative Models
• Rank-One Network: An Effective Framework for Image Restoration
• Image Restoration for Under-Display Camera
• Learning Many-to-Many Mapping for Unpaired Real-World Image Super-resolution and Downscaling
• Image Processing Using Multi-Code GAN Prior
• Rethinking Class-Discrimination Based CNN Channel Pruning
• TransGaGa: Geometry-Aware Unsupervised Image-to-Image Translation
• PI-REC: Progressive Image Reconstruction Network With Edge and Color Domain
• Multi-Attention Based Ultra Lightweight Image Super-Resolution
• Collapsible Linear Blocks for Super-Efficient Super Resolution
• Spatial-Angular Interaction for Light Field Image Super-Resolution
• EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning
• Effectiveness of State-of-the-Art Super Resolution Algorithms in Surveillance Environment
• Component Divide-and-Conquer for Real-World Image Super-Resolution
• GazeCorrection:Self-Guided Eye Manipulation in the wild using Self-Supervised Generative Adversarial Networks
• Progressive Perception-Oriented Network for Single Image Super-Resolution
• Perceptual Extreme Super Resolution Network with Receptive Field Block
• Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
• Spectrum-to-Kernel Translation for Accurate Blind Image Super-Resolution
• Co-VeGAN: Complex-Valued Generative Adversarial Network for Compressive Sensing MR Image Reconstruction
• A Theory of the Distortion-Perception Tradeoff in Wasserstein Space
• SeG-SR: Integrating Semantic Knowledge into Remote Sensing Image Super-Resolution via Vision-Language Model
• Unpaired Image Super-Resolution using Pseudo-Supervision
• Ship in Sight: Diffusion Models for Ship-Image Super Resolution
• KOALAnet: Blind Super-Resolution using Kernel-Oriented Adaptive Local Adjustment
• A new method for structural diagnostics with muon tomography and deep learning
• On Perceptual Lossy Compression: The Cost of Perceptual Reconstruction and An Optimal Training Framework
• Machine-learned 3D Building Vectorization from Satellite Imagery
• Attention-Guided Multi-scale Interaction Network for Face Super-Resolution
• Attaining Real-Time Super-Resolution for Microscopic Images Using GAN
• Neural Network Libraries: A Deep Learning Framework Designed from Engineers' Perspectives
• CFSNet: Toward a Controllable Feature Space for Image Restoration
• Approaching the Limit of Image Rescaling via Flow Guidance
• Super-Resolution Perception for Industrial Sensor Data
• Towards Compact Single Image Super-Resolution via Contrastive Self-distillation
• Unsupervised Super-Resolution of Satellite Imagery for High Fidelity Material Label Transfer
• EvTexture++: Event-Driven Texture Enhancement for Video Super-Resolution
• Modulating Image Restoration with Continual Levels via Adaptive Feature Modification Layers
• ReLGAN: Generalization of Consistency for GAN with Disjoint Constraints and Relative Learning of Generative Processes for Multiple Transformation Learning
• A Matrix-in-matrix Neural Network for Image Super Resolution
• A Progressive Image Restoration Network for High-order Degradation Imaging in Remote Sensing
• Deep Network Interpolation for Continuous Imagery Effect Transition
• JSI-GAN: GAN-Based Joint Super-Resolution and Inverse Tone-Mapping with Pixel-Wise Task-Specific Filters for UHD HDR Video
• Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution
• Optimizing Generative Adversarial Networks for Image Super Resolution via Latent Space Regularization
• First-order State Space Model for Lightweight Image Super-resolution
• Solving the inverse problem of microscopy deconvolution with a residual Beylkin-Coifman-Rokhlin neural network
• Landmark Detection in Low Resolution Faces with Semi-Supervised Learning
• Symmetric Parallax Attention for Stereo Image Super-Resolution
• Unified Dynamic Convolutional Network for Super-Resolution with Variational Degradations
• NODE: Extreme Low Light Raw Image Denoising using a Noise Decomposition Network
• Fine-grained Attention and Feature-sharing Generative Adversarial Networks for Single Image Super-Resolution
• FAN: Frequency Aggregation Network for Real Image Super-resolution
• MIASSR: An Approach for Medical Image Arbitrary Scale Super-Resolution
• Image-to-Image Translation with Low Resolution Conditioning
• Bringing Old Photos Back to Life
• A comparative study of various Deep Learning techniques for spatio-temporal Super-Resolution reconstruction of Forced Isotropic Turbulent flows
• Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions
• Learned Block-based Hybrid Image Compression
• Multi-level Wavelet-based Generative Adversarial Network for Perceptual Quality Enhancement of Compressed Video
• Training Generative Adversarial Networks in One Stage
• MSN: Efficient Online Mask Selection Network for Video Instance Segmentation
• OnlyFlow: Optical Flow based Motion Conditioning for Video Diffusion Models
• Equivariant Imaging: Learning Beyond the Range Space
• PCA-SRGAN: Incremental Orthogonal Projection Discrimination for Face Super-resolution
• Semantic Is Enough: Only Semantic Information For NeRF Reconstruction
• Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution
• Dynamic Attention-Guided Diffusion for Image Super-Resolution
• Auto-Embedding Generative Adversarial Networks for High Resolution Image Synthesis
• Self-supervised Fine-tuning for Correcting Super-Resolution Convolutional Neural Networks
• Towards A Multi-agent System for Online Hate Speech Detection
• Quaternion Wavelet-Conditioned Diffusion Models for Image Super-Resolution
• Generative Design of Hardware-aware DNNs
• Residual Channel Attention Generative Adversarial Network for Image Super-Resolution and Noise Reduction
• Invertible Image Rescaling
• Estimates of maize plant density from UAV RGB images using Faster-RCNN detection model: impact of the spatial resolution
• Learning the Loss Functions in a Discriminative Space for Video Restoration
• Hierarchical Back Projection Network for Image Super-Resolution
• SRZoo: An integrated repository for super-resolution using deep learning
• Deep Feedback Inverse Problem Solver
• A Survey on GAN Acceleration Using Memory Compression Technique
• RankSRGAN: Super Resolution Generative Adversarial Networks with Learning to Rank
• Noise Conditional Flow Model for Learning the Super-Resolution Space
• Human Vision Constrained Super-Resolution
• FireSRnet: Geoscience-Driven Super-Resolution of Future Fire Risk from Climate Change
• MIINet: An Image Quality Improvement Framework for Supporting Medical Diagnosis
• VSpSR: Explorable Super-Resolution via Variational Sparse Representation
• Deep learning of multi-resolution X-Ray micro-CT images for multi-scale modelling
• A Two-stage Deep Network for High Dynamic Range Image Reconstruction
• Region-Adaptive Deformable Network for Image Quality Assessment
• Feature Super-Resolution Based Facial Expression Recognition for Multi-scale Low-Resolution Faces
• Joint Super-Resolution and Rectification for Solar Cell Inspection
• Implicit Subspace Prior Learning for Dual-Blind Face Restoration
• Boosting High-Level Vision with Joint Compression Artifacts Reduction and Super-Resolution
• Disentangle Perceptual Learning through Online Contrastive Learning
• HypervolGAN: An efficient approach for GAN with multi-objective training function
• Reconstruct high-resolution multi-focal plane images from a single 2D wide field image
• Image Super-Resolution via Attention based Back Projection Networks
• Nested Scale Editing for Conditional Image Synthesis
• Regularized Adaptation for Stable and Efficient Continuous-Level Learning on Image Processing Networks
• Zoom in to the details of human-centric videos
• Multi-scale Grouped Dense Network for VVC Intra Coding
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Neural Enhancement in Content Delivery Systems: The State-of-the-Art and Future Directions
• Editorial: Introduction to the Issue on Deep Learning for Image/Video Restoration and Compression
• Multimodal Image Synthesis with Conditional Implicit Maximum Likelihood Estimation
• A Frequency Domain Constraint for Synthetic and Real X-ray Image Super Resolution
• Enhanced MRI Reconstruction Network using Neural Architecture Search
• Zoom, Enhance! Measuring Surveillance GAN Up-sampling
• Exploiting Aliasing for Manga Restoration
• RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network
• Learning Structral coherence Via Generative Adversarial Network for Single Image Super-Resolution
• DEM Super-Resolution with EfficientNetV2
• Joint Demosaicing and Super-Resolution (JDSR): Network Design and Perceptual Optimization
• A Reduced Codebook and Re-Interpolation Approach for Enhancing Quality in Chroma Subsampling
• Frequency Consistent Adaptation for Real World Super Resolution
• Separating Content and Style for Unsupervised Image-to-Image Translation
• Multi-FAN: Multi-Spectral Mosaic Super-Resolution Via Multi-Scale Feature Aggregation Network
• Multimodal Image Outpainting With Regularized Normalized Diversification
• Legacy Photo Editing with Learned Noise Prior
• Deep priors for satellite image restoration with accurate uncertainties
• Deep Snapshot HDR Imaging Using Multi-Exposure Color Filter Array
• Real-time Surgical Environment Enhancement for Robot-Assisted Minimally Invasive Surgery Based on Super-Resolution
• Capsule GAN for Prostate MRI Super-Resolution
• Cascading Modular Network (CAM-Net) for Multimodal Image Synthesis
• Debiased Subjective Assessment of Real-World Image Enhancement
• Zoom To Learn, Learn To Zoom
• Deep Space-Time Video Upsampling Networks
• Video Super-Resolution with Long-Term Self-Exemplars
• Functional Neural Networks for Parametric Image Restoration Problems
• NTIRE 2020 Challenge on Video Quality Mapping: Methods and Results