Video Frame Interpolation via Adaptive Separable Convolution

Abstract

Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent approaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand. To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation.

ICCV 2017, http://graphics.cs.pdx.edu/project/sepconv/

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Deep Convolutional Inverse Graphics Network
• Video (language) modeling: a baseline for generative models of natural videos
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Compression Artifacts Removal Using Convolutional Neural Networks
• Generative Visual Manipulation on the Natural Image Manifold
• Learning a Convolutional Neural Network for Non-uniform Motion Blur Removal
• Video Frame Interpolation via Adaptive Convolution
• Learning to Extract Motion from Videos in Convolutional Neural Networks

Cited by in corpus (20)

• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Image and Video Compression with Neural Networks: A Review
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Stochastic Adversarial Video Prediction
• Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
• Context-aware Synthesis for Video Frame Interpolation
• Spatio-Temporal Filter Adaptive Network for Video Deblurring
• Video Compression through Image Interpolation
• Deep Online Video Stabilization
• Temporal Interpolation via Motion Field Prediction
• Super-Resolution with Deep Adaptive Image Resampling
• Few-shot Video-to-Video Synthesis
• Video Frame Interpolation by Plug-and-Play Deep Locally Linear Embedding
• JSI-GAN: GAN-Based Joint Super-Resolution and Inverse Tone-Mapping with Pixel-Wise Task-Specific Filters for UHD HDR Video
• A Temporally-Aware Interpolation Network for Video Frame Inpainting
• Temporal Interpolation as an Unsupervised Pretraining Task for Optical Flow Estimation
• Implementing Adaptive Separable Convolution for Video Frame Interpolation
• DAN: A Deformation-Aware Network for Consecutive Biomedical Image Interpolation
• Learning Light Field Angular Super-Resolution via a Geometry-Aware Network
• Learned Quality Enhancement via Multi-Frame Priors for HEVC Compliant Low-Delay Applications