Random deep neural networks are biased towards simple functions

Abstract

We prove that the binary classifiers of bit strings generated by random wide deep neural networks with ReLU activation function are biased towards simple functions. The simplicity is captured by the following two properties. For any given input bit string, the average Hamming distance of the closest input bit string with a different classification is at least sqrt(n / (2π log n)), where n is the length of the string. Moreover, if the bits of the initial string are flipped randomly, the average number of flips required to change the classification grows linearly with n. These results are confirmed by numerical experiments on deep neural networks with two hidden layers, and settle the conjecture stating that random deep neural networks are biased towards simple functions. This conjecture was proposed and numerically explored in [Valle Pérez et al., ICLR 2019] to explain the unreasonably good generalization properties of deep learning algorithms. The probability distribution of the functions generated by random deep neural networks is a good choice for the prior probability distribution in the PAC-Bayesian generalization bounds. Our results constitute a fundamental step forward in the characterization of this distribution, therefore contributing to the understanding of the generalization properties of deep learning algorithms.

Cited by in corpus (13)

• Machine Learning and Deep Learning -- A review for Ecologists
• Boundary thickness and robustness in learning models
• Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning
• On the infinite width limit of neural networks with a standard parameterization
• Neural networks are a priori biased towards Boolean functions with low entropy
• Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective
• Batch Normalization Orthogonalizes Representations in Deep Random Networks
• Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks
• Is SGD a Bayesian sampler? Well, almost
• The shape and simplicity biases of adversarially robust ImageNet-trained CNNs
• Adversarial Robustness Guarantees for Random Deep Neural Networks
• Initializing ReLU networks in an expressive subspace of weights
• On the Power of Deep but Naive Partial Label Learning