LDMVFI: Video Frame Interpolation with Latent Diffusion Models
arXiv:2303.09508 · doi:10.1609/aaai.v38i2.27912
Abstract
Existing works on video frame interpolation (VFI) mostly employ deep neural networks that are trained by minimizing the L1, L2, or deep feature space distance (e.g. VGG loss) between their outputs and ground-truth frames. However, recent works have shown that these metrics are poor indicators of perceptual VFI quality. Towards developing perceptually-oriented VFI methods, in this work we propose latent diffusion model-based VFI, LDMVFI. This approaches the VFI problem from a generative perspective by formulating it as a conditional generation problem. As the first effort to address VFI using latent diffusion models, we rigorously benchmark our method on common test sets used in the existing VFI literature. Our quantitative experiments and user study indicate that LDMVFI is able to interpolate video content with favorable perceptual quality compared to the state of the art, even in the high-resolution regime. Our code is available at https://github.com/danier97/LDMVFI.
arXiv commentAccepted by AAAI 2024
References in corpus (33)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Auto-Encoding Variational Bayes
- Distilling the Knowledge in a Neural Network
- Decoupled Weight Decay Regularization
- Denoising Diffusion Probabilistic Models
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Neural Discrete Representation Learning
- Diffusion Models Beat GANs on Image Synthesis
- Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- High-Resolution Image Synthesis with Latent Diffusion Models
- Cascaded Diffusion Models for High Fidelity Image Generation
- Elucidating the Design Space of Diffusion-Based Generative Models
- DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
- Progressive Distillation for Fast Sampling of Diffusion Models
- ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation
- MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
- FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
- Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
- MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement
- Context-aware Synthesis for Video Frame Interpolation
- Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolution
- A Subjective Quality Study for Video Frame Interpolation
- BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation
- Enhancing Deformable Convolution based Video Frame Interpolation with Coarse-to-fine 3D CNN
- IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation
- CDFI: Compression-Driven Network Design for Frame Interpolation
- Perceptual Learned Video Compression with Recurrent Conditional GAN
- Asymmetric Bilateral Motion Estimation for Video Frame Interpolation
- NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Dataset and Study
Cited by in corpus (4)
- Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization
- Reversing the Damage: A QP-Aware Transformer-Diffusion Approach for 8K Video Restoration under Codec Compression
- RMT-BVQA: Recurrent Memory Transformer-based Blind Video Quality Assessment for Enhanced Video Content
- Learning to Refocus with Video Diffusion Models