Low-Latency Video Semantic Segmentation
arXiv:1804.00389
Abstract
Recent years have seen remarkable progress in semantic segmentation. Yet, it remains a challenging task to apply segmentation techniques to video-based applications. Specifically, the high throughput of video streams, the sheer cost of running fully convolutional networks, together with the low-latency requirements in many real-world applications, e.g. autonomous driving, present a significant challenge to the design of the video segmentation framework. To tackle this combined challenge, we develop a framework for video semantic segmentation, which incorporates two novel components: (1) a feature propagation module that adaptively fuses features over time via spatially variant convolution, thus reducing the cost of per-frame computation; and (2) an adaptive scheduler that dynamically allocate computation based on accuracy prediction. Both components work together to ensure low latency while maintaining high segmentation quality. On both Cityscapes and CamVid, the proposed framework obtained competitive performance compared to the state of the art, while substantially reducing the latency, from 360 ms to 119 ms.
arXiv commentAccepted by CVPR 2018 as Spotlight
References in corpus (14)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Residual Learning for Image Recognition
- Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
- ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Learning Deconvolution Network for Semantic Segmentation
- FlowNet: Learning Optical Flow with Convolutional Networks
- Pyramid Scene Parsing Network
- Semantic Image Segmentation via Deep Parsing Network
- Attention to Scale: Scale-aware Semantic Image Segmentation
- Learning Video Object Segmentation from Static Images
- STFCN: Spatio-Temporal FCN for Semantic Video Segmentation
- Semantic Video Segmentation by Gated Recurrent Flow Propagation
- Multi-Path Feedback Recurrent Neural Network for Scene Parsing