The Probabilistic Fault Tolerance of Neural Networks in the Continuous Limit
arXiv:1902.01686
Abstract
The loss of a few neurons in a brain rarely results in any visible loss of function. However, the insight into what "few" means in this context is unclear. How many random neuron failures will it take to lead to a visible loss of function? In this paper, we address the fundamental question of the impact of the crash of a random subset of neurons on the overall computation of a neural network and the error in the output it produces. We study fault tolerance of neural networks subject to small random neuron/weight crash failures in a probabilistic setting. We give provable guarantees on the robustness of the network to these crashes. Our main contribution is a bound on the error in the output of a network under small random Bernoulli crashes proved by using a Taylor expansion in the continuous limit, where close-by neurons at a layer are similar. The failure mode we adopt in our model is characteristic of neuromorphic hardware, a promising technology to speed up artificial neural networks, as well as of biological networks. We show that our theoretical bounds can be used to compare the fault tolerance of different architectures and to design a regularizer improving the fault tolerance of a given architecture. We design an algorithm achieving fault tolerance using a reasonable number of neurons. In addition to the theoretical proof, we also provide experimental validation of our results and suggest a connection to the generalization capacity problem.
arXiv comment10 pages (without references), 2 figures, 2 tables, 1 algorithm, 26 pages of supplementary material
References in corpus (7)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Vowel recognition with four coupled spin-torque nano-oscillators
- Adversarial Examples Are Not Bugs, They Are Features
- On Lazy Training in Differentiable Programming
- Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
- When Neurons Fail
- Deep Function Machines: Generalized Neural Networks for Topological Layer Expression