Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

Abstract

Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods.

CVPR 2016 paper with updated affiliations and supplemental material, fixed typo in equation 4

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
• Fully Convolutional Networks for Semantic Segmentation
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Deep Networks for Image Super-Resolution with Sparse Prior

Cited by in corpus (17)

• Parallel Multiscale Autoregressive Density Estimation
• AUNet: Attention-guided dense-upsampling networks for breast mass segmentation in whole mammograms
• Densely Connected High Order Residual Network for Single Frame Image Super Resolution
• AI-assisted super-resolution cosmological simulations III: Time evolution
• Image Comes Dancing with Collaborative Parsing-Flow Video Synthesis
• High-throughput, high-resolution registration-free generated adversarial network microscopy
• Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications
• Deep Tensor Convolution on Multicores
• GUN: Gradual Upsampling Network for Single Image Super-Resolution
• BAM: A Balanced Attention Mechanism for Single Image Super Resolution
• Pyramid Correlation based Deep Hough Voting for Visual Object Tracking
• Two-stage Visual Cues Enhancement Network for Referring Image Segmentation
• SRWarp: Generalized Image Super-Resolution under Arbitrary Transformation
• CALC2.0: Combining Appearance, Semantic and Geometric Information for Robust and Efficient Visual Loop Closure
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
• Video Super-Resolution with Long-Term Self-Exemplars
• Sparse to Dense Motion Transfer for Face Image Animation