Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations

Abstract

Rectified Linear Units (ReLU) have become the main model for the neural units in current deep learning systems. This choice has been originally suggested as a way to compensate for the so called vanishing gradient problem which can undercut stochastic gradient descent (SGD) learning in networks composed of multiple layers. Here we provide analytical results on the effects of ReLUs on the capacity and on the geometrical landscape of the solution space in two-layer neural networks with either binary or real-valued weights. We study the problem of storing an extensive number of random patterns and find that, quite unexpectedly, the capacity of the network remains finite as the number of neurons in the hidden layer increases, at odds with the case of threshold units in which the capacity diverges. Possibly more important, a large deviation approach allows us to find that the geometrical landscape of the solution space has a peculiar structure: while the majority of solutions are close in distance but still isolated, there exist rare regions of solutions which are much more dense than the similar ones in the case of threshold units. These solutions are robust to perturbations of the weights and can tolerate large perturbations of the inputs. The analytical results are corroborated by numerical findings.

11 pages, 3 figures

References in corpus (8)

• Recipes for metastable states in Spin Glasses
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Shaping the learning landscape in neural networks around wide flat minima
• Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
• Origin of the computational hardness for learning with binary synapses
• Activation function dependence of the storage capacity of treelike neural networks
• A method to reduce the rejection rate in Monte Carlo Markov Chains

Cited by in corpus (21)

• Shaping the learning landscape in neural networks around wide flat minima
• Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation
• Drawing Phase Diagrams of Random Quantum Systems by Deep Learning the Wave Functions
• Unveiling the structure of wide flat minima in neural networks
• Learning through atypical "phase transitions" in overparameterized neural networks
• Entropic gradient descent algorithms and wide flat minima
• Model-Based Policy Search Using Monte Carlo Gradient Estimation with Real Systems Application
• Statistical learning theory of structured data
• Counting the learnable functions of structured data
• Activation function dependence of the storage capacity of treelike neural networks
• Clustering of solutions in the symmetric binary perceptron
• Efficiency of local learning rules in threshold-linear associative networks
• Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
• Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks
• On neural network kernels and the storage capacity problem
• Partial local entropy and anisotropy in deep weight spaces
• On the topology of solutions to random continuous constraint satisfaction problems
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation