A Single Video Super-Resolution GAN for Multiple Downsampling Operators based on Pseudo-Inverse Image Formation Models

Abstract

The popularity of high and ultra-high definition displays has led to the need for methods to improve the quality of videos already obtained at much lower resolutions. Current Video Super-Resolution methods are not robust to mismatch between training and testing degradation models since they are trained against a single degradation model (usually bicubic downsampling). This causes their performance to deteriorate in real-life applications. At the same time, the use of only the Mean Squared Error during learning causes the resulting images to be too smooth. In this work we propose a new Convolutional Neural Network for video super resolution which is robust to multiple degradation models. During training, which is performed on a large dataset of scenes with slow and fast motions, it uses the pseudo-inverse image formation model as part of the network architecture in conjunction with perceptual losses, in addition to a smoothness constraint that eliminates the artifacts originating from these perceptual losses. The experimental validation shows that our approach outperforms current state-of-the-art methods and is robust to multiple degradations.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss
• Image Super-Resolution Using Very Deep Residual Channel Attention Networks
• Generative Adversarial Networks and Perceptual Losses for Video Super-Resolution
• Amortised MAP Inference for Image Super-resolution
• Learning Deep CNN Denoiser Prior for Image Restoration
• Detail-revealing Deep Video Super-resolution
• Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation