Neural networks are a priori biased towards Boolean functions with low entropy

Abstract

Understanding the inductive bias of neural networks is critical to explaining their ability to generalise. Here, for one of the simplest neural networks -- a single-layer perceptron with n input neurons, one output neuron, and no threshold bias term -- we prove that upon random initialisation of weights, the a priori probability $P(t)$ that it represents a Boolean function that classifies t points in ${0,1}^n$ as 1 has a remarkably simple form: $P(t)=2^{-n}$ for $0\le t<2^n$. Since a perceptron can express far fewer Boolean functions with small or large values of t (low entropy) than with intermediate values of t (high entropy) there is, on average, a strong intrinsic a-priori bias towards individual functions with low entropy. Furthermore, within a class of functions with fixed t, we often observe a further intrinsic bias towards functions of lower complexity. Finally, we prove that, regardless of the distribution of inputs, the bias towards low entropy becomes monotonically stronger upon adding ReLU layers, and empirically show that increasing the variance of the bias term has a similar effect.

References in corpus (12)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Neural Networks as Gaussian Processes
• Exponential expressivity in deep neural networks through transient chaos
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• On the Spectral Bias of Neural Networks
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Deep Convolutional Networks as shallow Gaussian Processes
• A Fine-Grained Spectral Perspective on Neural Networks
• Random deep neural networks are biased towards simple functions
• Sample Variance Decay in Randomly Initialized ReLU Networks

Cited by in corpus (4)

• Is SGD a Bayesian sampler? Well, almost
• Deep frequency principle towards understanding why deeper learning is faster
• Why flatness does and does not correlate with generalization for deep neural networks
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networks