MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement

Abstract

Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets.

arXiv commentTo appear in IEEE Transactions on Pattern Analysis and Machine Intelligence

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• DeepStereo: Learning to Predict New Views from the World's Imagery
• Frame Interpolation with Multi-Scale Deep Loss Functions and Generative Adversarial Networks

Cited by in corpus (8)

• LDMVFI: Video Frame Interpolation with Latent Diffusion Models
• Neighbor Correspondence Matching for Flow-based Video Frame Synthesis
• All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling
• Space-Time-Aware Multi-Resolution Video Enhancement
• Revisiting Adaptive Convolutions for Video Frame Interpolation
• A Dual Camera System for High Spatiotemporal Resolution Video Acquisition
• Softmax Splatting for Video Frame Interpolation
• AIM 2019 Challenge on Video Temporal Super-Resolution: Methods and Results