Real-time Semantic Image Segmentation via Spatial Sparsity

Abstract

We propose an approach to semantic (image) segmentation that reduces the computational costs by a factor of 25 with limited impact on the quality of results. Semantic segmentation has a number of practical applications, and for most such applications the computational costs are critical. The method follows a typical two-column network structure, where one column accepts an input image, while the other accepts a half-resolution version of that image. By identifying specific regions in the full-resolution image that can be safely ignored, as well as carefully tailoring the network structure, we can process approximately 15 highresolution Cityscapes images (1024x2048) per second using a single GTX 980 video card, while achieving a mean intersection-over-union score of 72.9% on the Cityscapes test set.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Pyramid Scene Parsing Network
• Bridging Category-level and Instance-level Semantic Image Segmentation
• Attention to Scale: Scale-aware Semantic Image Segmentation
• ICNet for Real-Time Semantic Segmentation on High-Resolution Images
• RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation
• Learning to Refine Object Segments
• Supervised Deep Sparse Coding Networks

Cited by in corpus (11)

• BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation
• DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation
• Real-Time Semantic Segmentation via Multiply Spatial Fusion Network
• Deep Multi-Branch Aggregation Network for Real-Time Semantic Segmentation in Street Scenes
• SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation
• Real-Time High-Performance Semantic Image Segmentation of Urban Street Scenes
• Real-time Progressive 3D Semantic Segmentation for Indoor Scene
• Real-Time Semantic Segmentation via Auto Depth, Downsampling Joint Decision and Feature Aggregation
• Associative Embedding for Game-Agnostic Team Discrimination
• TapLab: A Fast Framework for Semantic Video Segmentation Tapping into Compressed-Domain Knowledge
• Dense Dual-Path Network for Real-time Semantic Segmentation