Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias

Abstract

Despite the phenomenal success of deep neural networks in a broad range of learning tasks, there is a lack of theory to understand the way they work. In particular, Convolutional Neural Networks (CNNs) are known to perform much better than Fully-Connected Networks (FCNs) on spatially structured data: the architectural structure of CNNs benefits from prior knowledge on the features of the data, for instance their translation invariance. The aim of this work is to understand this fact through the lens of dynamics in the loss landscape. We introduce a method that maps a CNN to its equivalent FCN (denoted as eFCN). Such an embedding enables the comparison of CNN and FCN training dynamics directly in the FCN space. We use this method to test a new training protocol, which consists in training a CNN, embedding it to FCN space at a certain ``relax time'', then resuming the training in FCN space. We observe that for all relax times, the deviation from the CNN subspace is small, and the final performance reached by the eFCN is higher than that reachable by a standard FCN of same architecture. More surprisingly, for some intermediate relax times, the eFCN outperforms the CNN it stemmed, by combining the prior information of the CNN and the expressivity of the FCN in a complementary way. The practical interest of our protocol is limited by the very large size of the highly sparse eFCN. However, it offers interesting insights into the persistence of architectural bias under stochastic gradient dynamics. It shows the existence of some rare basins in the FCN loss landscape associated with very good generalization. These can only be accessed thanks to the CNN prior, which helps navigate the landscape during the early stages of optimization.

arXiv commentUpdate for the camera ready version - NeurIPS 2019

References in corpus (23)

• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Understanding deep learning requires rethinking generalization
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Three Factors Influencing Minima in SGD
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Essentially No Barriers in Neural Network Energy Landscape
• Convolutional Neural Fabrics
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Scaling description of generalization with number of parameters in deep learning
• Net2Net: Accelerating Learning via Knowledge Transfer
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Shaping the learning landscape in neural networks around wide flat minima
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Critical Learning Periods in Deep Neural Networks
• Generalization bounds for deep convolutional neural networks
• Homotopy Analysis for Tensor PCA

Cited by in corpus (1)

• ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases