Effect of Architectures and Training Methods on the Performance of Learned Video Frame Prediction
arXiv:2008.06106 · doi:10.1109/ICIP.2019.8803624
Abstract
We analyze the performance of feedforward vs. recurrent neural network (RNN) architectures and associated training methods for learned frame prediction. To this effect, we trained a residual fully convolutional neural network (FCNN), a convolutional RNN (CRNN), and a convolutional long short-term memory (CLSTM) network for next frame prediction using the mean square loss. We performed both stateless and stateful training for recurrent networks. Experimental results show that the residual FCNN architecture performs the best in terms of peak signal to noise ratio (PSNR) at the expense of higher training and test (inference) computational complexity. The CRNN can be trained stably and very efficiently using the stateful truncated backpropagation through time procedure, and it requires an order of magnitude less inference runtime to achieve near real-time frame prediction with an acceptable performance.
arXiv commentAccepted for publication at IEEE ICIP 2019
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- On the difficulty of training Recurrent Neural Networks
- Unsupervised Learning of Video Representations using LSTMs
- Stochastic Video Generation with a Learned Prior
- Stochastic Adversarial Video Prediction
- Learning to Generate Long-term Future via Hierarchical Prediction
- Hierarchical Long-term Video Prediction without Supervision
- Stochastic Variational Video Prediction
- Transformation-Based Models of Video Sequences
- Video Ladder Networks
- Deep Learned Frame Prediction for Video Compression