Activation function dependence of the storage capacity of treelike neural networks

Abstract

The expressive power of artificial neural networks crucially depends on the nonlinearity of their activation functions. Though a wide variety of nonlinear activation functions have been proposed for use in artificial neural networks, a detailed understanding of their role in determining the expressive power of a network has not emerged. Here, we study how activation functions affect the storage capacity of treelike two-layer networks. We relate the boundedness or divergence of the capacity in the infinite-width limit to the smoothness of the activation function, elucidating the relationship between previously studied special cases. Our results show that nonlinearity can both increase capacity and decrease the robustness of classification, and provide simple estimates for the capacity of networks with several commonly used activation functions. Furthermore, they generate a hypothesis for the functional benefit of dendritic spikes in branched neurons.

arXiv comment5+23 pages, 2+4 figures. v3: accepted for publication as a Letter in Physical Review E

References in corpus (9)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Perceptron capacity revisited: classification ability for correlated patterns
• The committee machine: Computational to statistical gaps in learning a two-layers neural network
• Statistical learning theory of structured data
• Statistical physics of neural systems with non-additive dendritic coupling
• Space of Functions Computed by Deep-Layered Machines

Cited by in corpus (6)

• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Unveiling the structure of wide flat minima in neural networks
• Contrasting random and learned features in deep Bayesian linear regression
• Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks
• On neural network kernels and the storage capacity problem
• Nonideality-Aware Training for Accurate and Robust Low-Power Memristive Neural Networks