Techniques of replica symmetry breaking and the storage problem of the McCulloch-Pitts neuron
arXiv:cond-mat/0010025 · doi:10.1016/S0370-1573(00)00073-9
Abstract
In this article the framework for Parisi's spontaneous replica symmetry breaking is reviewed, and subsequently applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron. The technical details are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and a Green function technique introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The ensuing graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and is shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them one similar to the Parisi phase in the Sherrington-Kirkpatrick model. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared to the prediction of the theory.
103 Latex pages (with REVTeX 3.0), including 15 figures (ps, epsi, eepic), accepted for Physics Reports
References in corpus (30)
- A Stochastic Tunneling Approach for Global Minimization of Complex Potential Energy Landscapes
- Evidence for the droplet/scaling picture of spin glasses
- Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
- Simplicity of State and Overlap Structure in Finite-Volume Realistic Spin Glasses
- Glassy transition in a disordered model for the RNA secondary structure
- Quantum description of spherical spins
- Replica Symmetry Breaking Instability in the 2D XY model in a random field
- Exactly solvable model of a quantum spin glass
- Matrix Games, Mixed Strategies, and Statistical Mechanics
- A general method to determine replica symmetry breaking transitions
- Statics, metastable states and barriers in protein folding: A replica variational approach
- Block diagonalizing ultrametric matrices
- Comment on "General Method to Determine Replica Symmetry Breaking Transitions"
- Multifractal Analysis of the Coupling Space of Feed-Forward Neural Networks
- Analytical and Numerical Study of Internal Representations in Multilayer Neural Networks with Binary Weights
- To maximize or not to maximize the free energy of glassy systems, !=?
- Replica Theory of Granular Media
- Parisi Phase in a Neuron
- Comment on "Evidence for the Droplet/Scaling Picture of Spin Glasses"
- Reply to "Comment on Evidence for the droplet picture of spin glasses"
- An exact learning algorithm for autoassociative neural networks with binary couplings
- Correlation of internal representations in feed-forward neural networks
- Multifractality and percolation in the coupling space of perceptrons
- Low-energy excitations in fermionic spin glasses: A quantum-dynamical image of Parisi symmetry breaking
- Beyond storage capacity in a single model neuron: Continuous replica symmetry breaking
- Storage capacity of correlated perceptrons
- Finite size scaling in neural networks
- Comment on "Finite size scaling in Neural Networks"
- Ginzburg-Landau theory of the cluster glass phase
- Correlations between hidden units in multilayer neural networks and replica symmetry breaking
Cited by in corpus (7)
- Evolutionary games on graphs
- Out-of-equilibrium dynamical mean-field equations for the perceptron model
- Critical jammed phase of the linear perceptron
- Critical energy landscape of linear soft spheres
- Evolving Stochastic Learning Algorithm Based on Tsallis Entropic Index
- Critical properties of the SAT/UNSAT transitions in the classification problem of structured data
- Statistical Physics of Feedforward Neural Networks