Future Segmentation Using 3D Structure
arXiv:1811.11358
Abstract
Predicting the future to anticipate the outcome of events and actions is a critical attribute of autonomous agents; particularly for agents which must rely heavily on real time visual data for decision making. Working towards this capability, we address the task of predicting future frame segmentation from a stream of monocular video by leveraging the 3D structure of the scene. Our framework is based on learnable sub-modules capable of predicting pixel-wise scene semantic labels, depth, and camera ego-motion of adjacent frames. We further propose a recurrent neural network based model capable of predicting future ego-motion trajectory as a function of a series of past ego-motion steps. Ultimately, we observe that leveraging 3D structure in the model facilitates successful prediction, achieving state of the art accuracy in future semantic segmentation.
References in corpus (9)
- Unsupervised Learning of Video Representations using LSTMs
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Xception: Deep Learning with Depthwise Separable Convolutions
- Video (language) modeling: a baseline for generative models of natural videos
- Unsupervised Learning of Depth and Ego-Motion from Video
- Long-term Planning by Short-term Prediction
- On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Training
- Predicting Deeper into the Future of Semantic Segmentation
- Video Pixel Networks