Understanding and mitigating noise in trained deep neural networks

Abstract

Deep neural networks unlocked a vast range of new applications by solving tasks of which many were previously deemed as reserved to higher human intelligence. One of the developments enabling this success was a boost in computing power provided by special purpose hardware, such as graphic or tensor processing units. However, these do not leverage fundamental features of neural networks like parallelism and analog state variables. Instead, they emulate neural networks relying on binary computing, which results in unsustainable energy consumption and comparatively low speed. Fully parallel and analogue hardware promises to overcome these challenges, yet the impact of analogue neuron noise and its propagation, i.e. accumulation, threatens rendering such approaches inept. Here, we determine for the first time the propagation of noise in deep neural networks comprising noisy nonlinear neurons in trained fully connected layers. We study additive and multiplicative as well as correlated and uncorrelated noise, and develop analytical methods that predict the noise level in any layer of symmetric deep neural networks or deep neural networks trained with back propagation. We find that noise accumulation is generally bound, and adding additional network layers does not worsen the signal to noise ratio beyond a limit. Most importantly, noise accumulation can be suppressed entirely when neuron activation functions have a slope smaller than unity. We therefore developed the framework for noise in fully connected deep neural networks implemented in analog systems, and identify criteria allowing engineers to design noise-resilient novel neural network hardware.

References in corpus (3)

• Deep Learning with Limited Numerical Precision
• Noise-Induced Synchronization and Clustering in Ensembles of Uncoupled Limit-Cycle Oscillators
• Optical Neural Networks: The 3D connection

Cited by in corpus (11)

• The physics of optical computing
• An optical neural network using less than 1 photon per multiplication
• Quantum-limited stochastic optical neural networks operating at a few quanta per activation
• Multiplexing-based control of stochastic resonance
• Noise mitigation strategies in physical feedforward neural networks
• Impact of white noise in artificial neural networks trained for classification: performance and noise mitigation strategies
• Multiplexing-based control of wavefront propagation: the interplay of inter-layer coupling, asymmetry and noise
• Impact of internal noise on convolutional neural networks
• Hybrid Quantum-inspired Resnet and Densenet for Pattern Recognition
• Internal noise in hardware deep and recurrent neural networks helps with learning
• Noise impact on recurrent neural network with linear activation function