Towards Learning Convolutions from Scratch

Abstract

Convolution is one of the most essential components of architectures used in computer vision. As machine learning moves towards reducing the expert bias and learning it from data, a natural next step seems to be learning convolution-like structures from scratch. This, however, has proven elusive. For example, current state-of-the-art architecture search algorithms use convolution as one of the existing modules rather than learning it from data. In an attempt to understand the inductive bias that gives rise to convolutions, we investigate minimum description length as a guiding principle and show that in some settings, it can indeed be indicative of the performance of architectures. To find architectures with small description length, we propose $\beta$-LASSO, a simple variant of LASSO algorithm that, when applied on fully-connected networks for image classification tasks, learns architectures with local connections and achieves state-of-the-art accuracies for training fully-connected nets on CIFAR-10 (85.19%), CIFAR-100 (59.56%) and SVHN (94.07%) bridging the gap between fully-connected and convolutional nets.

18 pages, 9 figures, 4 tables

References in corpus (5)

• Improving neural networks by preventing co-adaptation of feature detectors
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Meta-Learning Symmetries by Reparameterization

Cited by in corpus (8)

• Review: Deep Learning in Electron Microscopy
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• Computational Separation Between Convolutional and Fully-Connected Networks
• Towards Biologically Plausible Convolutional Networks
• Hierarchically Compositional Tasks and Deep Convolutional Networks
• Programming with Neural Surrogates of Programs
• The Effects of Image Distribution and Task on Adversarial Robustness
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?