Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform

Abstract

Despite that convolutional neural networks (CNN) have recently demonstrated high-quality reconstruction for single-image super-resolution (SR), recovering natural and realistic texture remains a challenging problem. In this paper, we show that it is possible to recover textures faithful to semantic classes. In particular, we only need to modulate features of a few intermediate layers in a single network conditioned on semantic segmentation probability maps. This is made possible through a novel Spatial Feature Transform (SFT) layer that generates affine transformation parameters for spatial-wise feature modulation. SFT layers can be trained end-to-end together with the SR network using the same loss function. During testing, it accepts an input image of arbitrary size and generates a high-resolution image with just a single forward pass conditioned on the categorical priors. Our final results show that an SR network equipped with SFT can generate more realistic and visually pleasing textures in comparison to state-of-the-art SRGAN and EnhanceNet.

This work is accepted in CVPR 2018. Our project page is http://mmlab.ie.cuhk.edu.hk/projects/SFTGAN/

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Fully Convolutional Networks for Semantic Segmentation
• A Learned Representation For Artistic Style
• Modulating early visual processing by language
• MemNet: A Persistent Memory Network for Image Restoration
• FiLM: Visual Reasoning with a General Conditioning Layer
• Deeply-Recursive Convolutional Network for Image Super-Resolution
• Be Your Own Prada: Fashion Synthesis with Structural Coherence
• Controlling Perceptual Factors in Neural Style Transfer

Cited by in corpus (20)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Deep Learning for Image Super-resolution: A Survey
• Super-Resolution via Image-Adapted Denoising CNNs: Incorporating External and Internal Learning
• EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
• Meta-SR: A Magnification-Arbitrary Network for Super-Resolution
• Unsupervised Degradation Learning for Single Image Super-Resolution
• Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning
• One-shot Face Reenactment
• Spatio-Temporal Filter Adaptive Network for Video Deblurring
• Channel-wise and Spatial Feature Modulation Network for Single Image Super-Resolution
• Unsupervised Learning of Monocular Depth Estimation with Bundle Adjustment, Super-Resolution and Clip Loss
• Dual Reconstruction Nets for Image Super-Resolution with Gradient Sensitive Loss
• Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications
• Resolution-invariant Person Re-Identification
• Deep Network Interpolation for Continuous Imagery Effect Transition
• Fine-grained Image-to-Image Transformation towards Visual Recognition
• Reconstructing High-resolution Turbulent Flows Using Physics-Guided Neural Networks
• The Unreasonable Effectiveness of Texture Transfer for Single Image Super-resolution
• SuperMeshing: A New Deep Learning Architecture for Increasing the Mesh Density of Metal Forming Stress Field with Attention Mechanism and Perceptual Features
• Learning Structral coherence Via Generative Adversarial Network for Single Image Super-Resolution