Revisiting Spatial Invariance with Low-Rank Local Connectivity

Abstract

Convolutional neural networks are among the most successful architectures in deep learning with this success at least partially attributable to the efficacy of spatial invariance as an inductive bias. Locally connected layers, which differ from convolutional layers only in their lack of spatial invariance, usually perform poorly in practice. However, these observations still leave open the possibility that some degree of relaxation of spatial invariance may yield a better inductive bias than either convolution or local connectivity. To test this hypothesis, we design a method to relax the spatial invariance of a network layer in a controlled manner; we create a \textit{low-rank} locally connected layer, where the filter bank applied at each position is constructed as a linear combination of basis set of filter banks with spatially varying combining weights. By varying the number of basis filter banks, we can control the degree of relaxation of spatial invariance. In experiments with small convolutional networks, we find that relaxing spatial invariance improves classification accuracy over both convolution and locally connected layers across MNIST, CIFAR-10, and CelebA datasets, thus suggesting that spatial invariance may be an overly restrictive prior.

References in corpus (13)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• Improving neural networks by preventing co-adaptation of feature detectors
• One weird trick for parallelizing convolutional neural networks
• DeepID3: Face Recognition with Very Deep Neural Networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
• CondConv: Conditionally Parameterized Convolutions for Efficient Inference
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Deep AutoRegressive Networks
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields
• Deciding How to Decide: Dynamic Routing in Artificial Neural Networks
• LatentGNN: Learning Efficient Non-local Relations for Visual Recognition

Cited by in corpus (2)

• Meta-Learning Symmetries by Reparameterization
• Learning compact generalizable neural representations supporting perceptual grouping