Pixel-wise Attentional Gating for Parsimonious Pixel Labeling

Abstract

To achieve parsimonious inference in per-pixel labeling tasks with a limited computational budget, we propose a \emph{Pixel-wise Attentional Gating} unit (\emph{PAG}) that learns to selectively process a subset of spatial locations at each layer of a deep convolutional network. PAG is a generic, architecture-independent, problem-agnostic mechanism that can be readily "plugged in" to an existing model with fine-tuning. We utilize PAG in two ways: 1) learning spatially varying pooling fields that improve model performance without the extra computation cost associated with multi-scale pooling, and 2) learning a dynamic computation policy for each pixel to decrease total computation while maintaining accuracy. We extensively evaluate PAG on a variety of per-pixel labeling tasks, including semantic segmentation, boundary detection, monocular depth and surface normal estimation. We demonstrate that PAG allows competitive or state-of-the-art performance on these tasks. Our experiments show that PAG learns dynamic spatial allocation of computation over the input image which provides better performance trade-offs compared to related approaches (e.g., truncating deep models or dynamically skipping whole layers). Generally, we observe PAG can reduce computation by $10\%$ without noticeable loss in accuracy and performance degrades gracefully when imposing stronger computational constraints.

arXiv commenthttps://www.ics.uci.edu/~skong2/PAG.html

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Categorical Reparameterization with Gumbel-Softmax
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• cuDNN: Efficient Primitives for Deep Learning
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Holistically-Nested Edge Detection
• Deep Networks with Stochastic Depth
• PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
• BlockDrop: Dynamic Inference Paths in Residual Networks

Cited by in corpus (12)

• Improving Semantic Segmentation via Video Propagation and Label Relaxation
• Improving Semantic Segmentation via Self-Training
• Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation
• Improved Techniques for Training Adaptive Deep Networks
• Deep attention-based classification network for robust depth prediction
• Image Reconstruction with Predictive Filter Flow
• Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images
• Deep Neural Network Perception Models and Robust Autonomous Driving Systems
• Unsupervised Monocular Depth Perception: Focusing on Moving Objects
• MOOD: Multi-level Out-of-distribution Detection
• Unsupervised Learning of Depth and Deep Representation for Visual Odometry from Monocular Videos in a Metric Space
• Exploring Temporal Information for Improved Video Understanding