Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks

Abstract

Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The parameter perturbations investigated include weight sparsification (dilution) and binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are more robust to parameter perturbations with respect to their counterparts with sign activation, which arguably is reflected in the simplicity of the functions they generate.

References in corpus (7)

• The large deviation approach to statistical mechanics
• Numerical computation of rare events via large deviation theory
• Random deep neural networks are biased towards simple functions
• Noisy Random Boolean Formulae - a Statistical Physics Perspective
• Exploring the Function Space of Deep-Learning Machines
• Computing with Noise - Phase Transitions in Boolean Formulas
• Random active path model of deep neural networks with diluted binary synapses

Cited by in corpus (6)

• Mean-field inference methods for neural networks
• Learning credit assignment
• Space of Functions Computed by Deep-Layered Machines
• Network reconstruction may not mean dynamics prediction
• Large and moderate deviations for Gaussian neural networks
• Understanding the stochastic dynamics of sequential decision-making processes: A path-integral analysis of multi-armed bandits