Photo-Realistic Video Prediction on Natural Videos of Largely Changing Frames
arXiv:2003.08635
Abstract
Recent advances in deep learning have significantly improved performance of video prediction. However, state-of-the-art methods still suffer from blurriness and distortions in their future predictions, especially when there are large motions between frames. To address these issues, we propose a deep residual network with the hierarchical architecture where each layer makes a prediction of future state at different spatial resolution, and these predictions of different layers are merged via top-down connections to generate future frames. We trained our model with adversarial and perceptual loss functions, and evaluated it on a natural video dataset captured by car-mounted cameras. Our model quantitatively outperforms state-of-the-art baselines in future frame prediction on video sequences of both largely and slightly changing frames. Furthermore, our model generates future frames with finer details and textures that are perceptually more realistic than the baselines, especially under fast camera motions.
arXiv comment16 pages, 7 figures
References in corpus (7)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Spectral Normalization for Generative Adversarial Networks
- Unsupervised Learning of Video Representations using LSTMs
- Pixel Recurrent Neural Networks
- Video (language) modeling: a baseline for generative models of natural videos
- Learning to Generate Long-term Future via Hierarchical Prediction
- Mixtures of conditional Gaussian scale mixtures applied to multiscale image representations