Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation

Abstract

Convolutional networks are not aware of an object's geometric variations, which leads to inefficient utilization of model and data capacity. To overcome this issue, recent works on deformation modeling seek to spatially reconfigure the data towards a common arrangement such that semantic recognition suffers less from deformation. This is typically done by augmenting static operators with learned free-form sampling grids in the image space, dynamically tuned to the data and task for adapting the receptive field. Yet adapting the receptive field does not quite reach the actual goal -- what really matters to the network is the "effective" receptive field (ERF), which reflects how much each pixel contributes. It is thus natural to design other approaches to adapt the ERF directly during runtime. In this work, we instantiate one possible solution as Deformable Kernels (DKs), a family of novel and generic convolutional operators for handling object deformations by directly adapting the ERF while leaving the receptive field untouched. At the heart of our method is the ability to resample the original kernel space towards recovering the deformation of objects. This approach is justified with theoretical insights that the ERF is strictly determined by data sampling locations and kernel values. We implement DKs as generic drop-in replacements of rigid kernels and conduct a series of empirical studies whose results conform with our theories. Over several tasks and standard base models, our approach compares favorably against prior works that adapt during runtime. In addition, further experiments suggest a working mechanism orthogonal and complementary to previous works.

Project page: http://people.eecs.berkeley.edu/~hangg/deformable-kernels/. Accepted as conference paper in ICLR 2020. First two authors contributed equally

References in corpus (14)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Group Equivariant Convolutional Networks
• Deformable Convolutional Networks
• Making Convolutional Networks Shift-Invariant Again
• Locally Scale-Invariant Convolutional Neural Networks
• Convolutional neural network architecture for geometric matching
• Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields
• Learning by Abstraction: The Neural State Machine
• Deformable Filter Convolution for Point Cloud Reasoning
• Dynamic Scale Inference by Entropy Minimization

Cited by in corpus (7)

• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• Dynamic Neural Networks: A Survey
• Reconfigurable Voxels: A New Representation for LiDAR-Based Point Clouds
• Dynamic Region-Aware Convolution
• Integrating Large Circular Kernels into CNNs through Neural Architecture Search
• One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems
• Sampling Equivariant Self-attention Networks for Object Detection in Aerial Images