Video Frame Interpolation via Adaptive Convolution

Abstract

Video frame interpolation typically involves two steps: motion estimation and pixel synthesis. Such a two-step approach heavily depends on the quality of motion estimation. This paper presents a robust video frame interpolation method that combines these two steps into a single process. Specifically, our method considers pixel synthesis for the interpolated frame as local convolution over two input frames. The convolution kernel captures both the local motion between the input frames and the coefficients for pixel synthesis. Our method employs a deep fully convolutional neural network to estimate a spatially-adaptive convolution kernel for each pixel. This deep neural network can be directly trained end to end using widely available video data without any difficult-to-obtain ground-truth data like optical flow. Our experiments show that the formulation of video interpolation as a single convolution process allows our method to gracefully handle challenges like occlusion, blur, and abrupt brightness change and enables high-quality video frame interpolation.

CVPR 2017, http://graphics.cs.pdx.edu/project/adaconv

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Convolutional Inverse Graphics Network
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Video (language) modeling: a baseline for generative models of natural videos
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Compression Artifacts Removal Using Convolutional Neural Networks
• Generative Visual Manipulation on the Natural Image Manifold
• Learning a Convolutional Neural Network for Non-uniform Motion Blur Removal
• Learning to Extract Motion from Videos in Convolutional Neural Networks

Cited by in corpus (11)

• Video Frame Interpolation via Adaptive Separable Convolution
• Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
• Dynamic Neural Networks: A Survey
• Context-aware Synthesis for Video Frame Interpolation
• PeerNets: Exploiting Peer Wisdom Against Adversarial Attacks
• Few-shot Video-to-Video Synthesis
• A Temporally-Aware Interpolation Network for Video Frame Inpainting
• DAN: A Deformation-Aware Network for Consecutive Biomedical Image Interpolation
• Deep CNN Frame Interpolation with Lessons Learned from Natural Language Processing
• InterpolationSLAM: A Novel Robust Visual SLAM System in Rotational Motion
• W-Cell-Net: Multi-frame Interpolation of Cellular Microscopy Videos