Data-driven emergence of convolutional structure in neural networks

Abstract

Exploiting data invariances is crucial for efficient learning in both artificial and biological neural circuits. Understanding how neural networks can discover appropriate representations capable of harnessing the underlying symmetries of their inputs is thus crucial in machine learning and neuroscience. Convolutional neural networks, for example, were designed to exploit translation symmetry and their capabilities triggered the first wave of deep learning successes. However, learning convolutions directly from translation-invariant data with a fully-connected network has so far proven elusive. Here, we show how initially fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs, resulting in localised, space-tiling receptive fields. These receptive fields match the filters of a convolutional network trained on the same task. By carefully designing data models for the visual scene, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs, which has long been recognised as the hallmark of natural images. We provide an analytical and numerical characterisation of the pattern-formation mechanism responsible for this phenomenon in a simple model and find an unexpected link between receptive field formation and tensor decomposition of higher-order input correlations. These results provide a new perspective on the development of low-level feature detectors in various sensory modalities, and pave the way for studying the impact of higher-order statistics on learning in neural networks.

arXiv commentMain text: 19 pages, 4 figures; Supplementary Material: 4 pages, 4 figures

References in corpus (22)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Geometric deep learning: going beyond Euclidean data
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Understanding Deep Convolutional Networks
• Group Equivariant Convolutional Networks
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• A mathematical theory of semantic development in deep neural networks
• TensorLy: Tensor Learning in Python
• Gauge Equivariant Convolutional Networks and the Icosahedral CNN
• Going in circles is the way forward: the role of recurrence in visual inference
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• On the localization transition in symmetric random matrices
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• Statistical and computational phase transitions in spiked tensor estimation
• Data-driven emergence of convolutional structure in neural networks
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
• Estimating Higher-Order Moments Using Symmetric Tensor Decomposition
• A spherical Hopfield model
• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
• On the Spectrum of Random Features Maps of High Dimensional Data

Cited by in corpus (8)

• A simple linear algebra identity to optimize Large-Scale Neural Network Quantum States
• Mapping of attention mechanisms to a generalized Potts model
• Data-driven emergence of convolutional structure in neural networks
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Gaussian Universality of Perceptrons with Random Labels
• Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation
• Observing Schrödinger's Cat with Artificial Intelligence: Emergent Classicality from Information Bottleneck
• Emergence of global receptive fields capturing multipartite quantum correlations