SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation
arXiv:2011.12025 · doi:10.1109/TPAMI.2022.3162528
Abstract
SegBlocks reduces the computational cost of existing neural networks, by dynamically adjusting the processing resolution of image regions based on their complexity. Our method splits an image into blocks and downsamples blocks of low complexity, reducing the number of operations and memory consumption. A lightweight policy network, selecting the complex regions, is trained using reinforcement learning. In addition, we introduce several modules implemented in CUDA to process images in blocks. Most important, our novel BlockPad module prevents the feature discontinuities at block borders of which existing methods suffer, while keeping memory consumption under control. Our experiments on Cityscapes, Camvid and Mapillary Vistas datasets for semantic segmentation show that dynamically processing images offers a better accuracy versus complexity trade-off compared to static baselines of similar complexity. For instance, our method reduces the number of floating-point operations of SwiftNet-RN18 by 60% and increases the inference speed by 50%, with only 0.3% decrease in mIoU accuracy on Cityscapes.
arXiv commentlong version, 12 pages. Accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence
References in corpus (23)
- Distilling the Knowledge in a Neural Network
- Categorical Reparameterization with Gumbel-Softmax
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- An Analysis of Deep Neural Network Models for Practical Applications
- Training Deep Neural Networks on Noisy Labels with Bootstrapping
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- DABNet: Depth-wise Asymmetric Bottleneck for Real-time Semantic Segmentation
- Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
- Non-local Neural Networks
- Conditional Computation in Neural Networks for faster models
- Channel Gating Neural Networks
- PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
- Deep Learning of Representations: Looking Forward
- Real-time Semantic Image Segmentation via Spatial Sparsity
- Batch-Shaping for Learning Conditional Channel Gated Networks
- Guided Upsampling Network for Real-Time Semantic Segmentation
- Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation
- Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
- Real-time Semantic Segmentation with Fast Attention
- Computation on Sparse Neural Networks: an Inspiration for Future Hardware