Inductive Bias of Deep Convolutional Networks through Pooling Geometry

Abstract

Our formal understanding of the inductive bias that drives the success of convolutional networks on computer vision tasks is limited. In particular, it is unclear what makes hypotheses spaces born from convolution and pooling operations so suitable for natural images. In this paper we study the ability of convolutional networks to model correlations among regions of their input. We theoretically analyze convolutional arithmetic circuits, and empirically validate our findings on other types of convolutional networks as well. Correlations are formalized through the notion of separation rank, which for a given partition of the input, measures how far a function is from being separable. We show that a polynomially sized deep network supports exponentially high separation ranks for certain input partitions, while being limited to polynomial separation ranks for others. The network's pooling geometry effectively determines which input partitions are favored, thus serves as a means for controlling the inductive bias. Contiguous pooling windows as commonly employed in practice favor interleaved partitions over coarse ones, orienting the inductive bias towards the statistics of natural images. Other pooling schemes lead to different preferences, and this allows tailoring the network to data that departs from the usual domain of natural imagery. In addition to analyzing deep networks, we show that shallow ones support only linear separation ranks, and by this gain insight into the benefit of functions brought forth by depth - they are able to efficiently model strong correlation under favored partitions of the input.

Published as a conference paper at ICLR 2017

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• SimNets: A Generalization of Convolutional Networks
• Tensorial Mixture Models

Cited by in corpus (17)

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Stabilizing the Lottery Ticket Hypothesis
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• Embedding Hard Physical Constraints in Neural Network Coarse-Graining of 3D Turbulence
• Generalization Error of Invariant Classifiers
• Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)
• Computational Separation Between Convolutional and Fully-Connected Networks
• Towards Robust Vision Transformer
• Sum-Product-Quotient Networks
• On the Blindspots of Convolutional Networks
• Learning with convolution and pooling operations in kernel methods
• On the Long-Term Memory of Deep Recurrent Networks
• Extreme Memorization via Scale of Initialization
• NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks
• Modelling Identity Rules with Neural Networks
• Resource Constrained Neural Networks for 5G Direction-of-Arrival Estimation in Micro-controllers
• A Closed-Form Learned Pooling for Deep Classification Networks