Mean Field Theory of Activation Functions in Deep Neural Networks
arXiv:1805.08786
Abstract
We present a Statistical Mechanics (SM) model of deep neural networks, connecting the energy-based and the feed forward networks (FFN) approach. We infer that FFN can be understood as performing three basic steps: encoding, representation validation and propagation. From the meanfield solution of the model, we obtain a set of natural activations -- such as Sigmoid, and ReLu -- together with the state-of-the-art, Swish; this represents the expected information propagating through the network and tends to ReLu in the limit of zero noise.We study the spectrum of the Hessian on an associated classification task, showing that Swish allows for more consistent performances over a wider range of network architectures.
arXiv commentPresented at the ICML 2019 Workshop on Theoretical Physics forDeep Learning
References in corpus (7)
- Opening the Black Box of Deep Neural Networks via Information
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Exponential expressivity in deep neural networks through transient chaos
- An exact mapping between the Variational Renormalization Group and Deep Learning
- Sharp Minima Can Generalize For Deep Nets
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- On the Selection of Initialization and Activation Function for Deep Neural Networks