3DSRnet: Video Super-resolution using 3D Convolutional Neural Networks

Abstract

In video super-resolution, the spatio-temporal coherence between, and among the frames must be exploited appropriately for accurate prediction of the high resolution frames. Although 2D convolutional neural networks (CNNs) are powerful in modelling images, 3D-CNNs are more suitable for spatio-temporal feature extraction as they can preserve temporal information. To this end, we propose an effective 3D-CNN for video super-resolution, called the 3DSRnet that does not require motion alignment as preprocessing. Our 3DSRnet maintains the temporal depth of spatio-temporal feature maps to maximally capture the temporally nonlinear characteristics between low and high resolution frames, and adopts residual learning in conjunction with the sub-pixel outputs. It outperforms the most state-of-the-art method with average 0.45 and 0.36 dB higher in PSNR for scales 3 and 4, respectively, in the Vidset4 benchmark. Our 3DSRnet first deals with the performance drop due to scene change, which is important in practice but has not been previously considered.

arXiv commentExtension of our paper accepted at ICIP 2019

Cited by in corpus (7)

• Revisiting Temporal Modeling for Video Super-resolution
• A Survey of Deep Learning Video Super-Resolution
• MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution
• VESR-Net: The Winning Solution to Youku Video Enhancement and Super-Resolution Challenge
• Video Super-resolution with Temporal Group Attention
• Multi-frame Joint Enhancement for Early Interlaced Videos
• Learnable Sampling 3D Convolution for Video Enhancement and Action Recognition