Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields

Abstract

The visual world is vast and varied, but its variations divide into structured and unstructured factors. We compose free-form filters and structured Gaussian filters, optimized end-to-end, to factorize deep representations and learn both local features and their degree of locality. Our semi-structured composition is strictly more expressive than free-form filtering, and changes in its structured parameters would require changes in free-form architecture. In effect this optimizes over receptive field size and shape, tuning locality to the data and task. Dynamic inference, in which the Gaussian structure varies with the input, adapts receptive field size to compensate for local scale variation. Optimizing receptive field size improves semantic segmentation accuracy on Cityscapes by 1-2 points for strong dilated and skip architectures and by up to 10 points for suboptimal designs. Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.

References in corpus (4)

• DARTS: Differentiable Architecture Search
• Fully Convolutional Networks for Semantic Segmentation
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Locally Scale-Invariant Convolutional Neural Networks

Cited by in corpus (10)

• Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation
• Scale-covariant and scale-invariant Gaussian derivative networks
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• Resolution learning in deep convolutional networks using scale-space theory
• Provably scale-covariant continuous hierarchical networks based on scale-normalized differential expressions coupled in cascade
• Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks
• Translation Insensitive CNNs
• Dynamic Scale Inference by Entropy Minimization
• Deep Hough-Transform Line Priors
• CoDeNet: Efficient Deployment of Input-Adaptive Object Detection on Embedded FPGAs