TDAN: Temporally Deformable Alignment Network for Video Super-Resolution

Abstract

Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames). Due to varying motion of cameras or objects, the reference frame and each support frame are not aligned. Therefore, temporal alignment is a challenging yet important problem for VSR. Previous VSR methods usually utilize optical flow between the reference frame and each supporting frame to wrap the supporting frame for temporal alignment. Therefore, the performance of these image-level wrapping-based models will highly depend on the prediction accuracy of optical flow, and inaccurate optical flow will lead to artifacts in the wrapped supporting frames, which also will be propagated into the reconstructed HR video frame. To overcome the limitation, in this paper, we propose a temporal deformable alignment network (TDAN) to adaptively align the reference frame and each supporting frame at the feature level without computing optical flow. The TDAN uses features from both the reference frame and each supporting frame to dynamically predict offsets of sampling convolution kernels. By using the corresponding kernels, TDAN transforms supporting frames to align with the reference frame. To predict the HR video frame, a reconstruction network taking aligned frames and the reference frame is utilized. Experimental results demonstrate the effectiveness of the proposed TDAN-based VSR model.

arXiv comment10 pages, 6 figures, demo link http://www.youtube.com/watch?v=eZExENE50I0

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Deep Residual Learning for Image Recognition
• Deformable Convolutional Networks
• Image Super-Resolution Using Very Deep Residual Channel Attention Networks
• Residual Dense Network for Image Super-Resolution
• MemNet: A Persistent Memory Network for Image Restoration
• Detail-revealing Deep Video Super-resolution
• Object Detection in Video with Spatiotemporal Sampling Networks
• Generalized Video Deblurring for Dynamic Scenes
• Integral Human Pose Regression
• Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

Cited by in corpus (8)

• Deformable Non-local Network for Video Super-Resolution
• EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
• Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution
• MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution
• Video Saliency Prediction Using Enhanced Spatiotemporal Alignment Network
• Deep Unrolled Network for Video Super-Resolution
• FineNet: Frame Interpolation and Enhancement for Face Video Deblurring
• Learnable Sampling 3D Convolution for Video Enhancement and Action Recognition