Zoom-In-to-Check: Boosting Video Interpolation via Instance-level Discrimination

Abstract

We propose a light-weight video frame interpolation algorithm. Our key innovation is an instance-level supervision that allows information to be learned from the high-resolution version of similar objects. Our experiment shows that the proposed method can generate state-of-the-art results across different datasets, with fractional computation resources (time and memory) of competing methods. Given two image frames, a cascade network creates an intermediate frame with 1) a flow-warping module that computes coarse bi-directional optical flow and creates an interpolated image via flow-based warping, followed by 2) an image synthesis module to make fine-scale corrections. In the learning stage, object detection proposals are generated on the interpolated image.Lower resolution objects are zoomed into, and the learning algorithms using an adversarial loss trained on high-resolution objects to guide the system towards the instance-level refinement corrects details of object shape and boundaries.

arXiv commentCVPR 2019 camera-ready, supplementary video: https://youtu.be/q-_wIRq26DY

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Spectral Normalization for Generative Adversarial Networks
• Generating Videos with Scene Dynamics
• Autoencoding beyond pixels using a learned similarity metric
• EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
• Stochastic Adversarial Video Prediction
• Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation