Multi-View Frame Reconstruction with Conditional GAN
arXiv:1809.10352 · doi:10.1109/GlobalSIP.2018.8646380
Abstract
Multi-view frame reconstruction is an important problem particularly when multiple frames are missing and past and future frames within the camera are far apart from the missing ones. Realistic coherent frames can still be reconstructed using corresponding frames from other overlapping cameras. We propose an adversarial approach to learn the spatio-temporal representation of the missing frame using conditional Generative Adversarial Network (cGAN). The conditional input to each cGAN is the preceding or following frames within the camera or the corresponding frames in other overlapping cameras, all of which are merged together using a weighted average. Representations learned from frames within the camera are given more weight compared to the ones learned from other cameras when they are close to the missing frames and vice versa. Experiments on two challenging datasets demonstrate that our framework produces comparable results with the state-of-the-art reconstruction method in a single camera and achieves promising performance in multi-camera scenario.
arXiv comment5 pages, 4 figures, 3 tables, Accepted at IEEE Global Conference on Signal and Information Processing, 2018
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Conditional Generative Adversarial Nets
- TensorFlow: A system for large-scale machine learning
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Generative Adversarial Text to Image Synthesis
- Unsupervised Learning of Video Representations using LSTMs
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- Video (language) modeling: a baseline for generative models of natural videos
- Video Inpainting of Complex Scenes
- Frame Interpolation with Multi-Scale Deep Loss Functions and Generative Adversarial Networks
- A Temporally-Aware Interpolation Network for Video Frame Inpainting