Value of Temporal Dynamics Information in Driving Scene Segmentation
arXiv:1904.00758
Abstract
Semantic scene segmentation has primarily been addressed by forming representations of single images both with supervised and unsupervised methods. The problem of semantic segmentation in dynamic scenes has begun to recently receive attention with video object segmentation approaches. What is not known is how much extra information the temporal dynamics of the visual scene carries that is complimentary to the information available in the individual frames of the video. There is evidence that the human visual system can effectively perceive the scene from temporal dynamics information of the scene's changing visual characteristics without relying on the visual characteristics of individual snapshots themselves. Our work takes steps to explore whether machine perception can exhibit similar properties by combining appearance-based representations and temporal dynamics representations in a joint-learning problem that reveals the contribution of each toward successful dynamic scene segmentation. Additionally, we provide the MIT Driving Scene Segmentation dataset, which is a large-scale full driving scene segmentation dataset, densely annotated for every pixel and every one of 5,000 video frames. This dataset is intended to help further the exploration of the value of temporal dynamics information for semantic segmentation in video.
References in corpus (12)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Rethinking Atrous Convolution for Semantic Image Segmentation
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
- Conditional Random Fields as Recurrent Neural Networks
- Pyramid Scene Parsing Network
- Fully Connected Deep Structured Networks
- The 2018 DAVIS Challenge on Video Object Segmentation
- Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
- Learning Video Object Segmentation with Visual Memory
- COCO-Stuff: Thing and Stuff Classes in Context
- YouTube-VOS: Sequence-to-Sequence Video Object Segmentation