Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings

Abstract

Energy-based probabilistic models learned by maximizing the likelihood of the data are limited by the intractability of the partition function. A widely used workaround is to maximize the pseudo-likelihood, which replaces the global normalization with tractable local normalizations. Here we show that, in the zero-temperature limit, a network trained to maximize pseudo-likelihood naturally implements an associative memory: if the training set is small, patterns become fixed-point attractors whose basins of attraction exceed those of any classical Hopfield rule. We explain quantitatively this effect on uncorrelated random patterns. Moreover, we show that, for different structured datasets coming from computer science (random feature model, MNIST), physics (spin glasses) and biology (proteins), as the number of training examples increases the learned network goes beyond memorization, developing meaningful attractors with non-trivial correlations with test examples, thus showing the ability to generalize. Our results therefore reveal pseudo-likelihood works both as an efficient inference tool and as a principled mechanism for memory and generalization.

References in corpus (19)

• Exchange Monte Carlo Method and Application to Spin Glass Simulations
• Improved contact prediction in proteins: Using pseudolikelihoods to infer Potts models
• Inverse statistical problems: from the inverse Ising problem to data science
• Inverse Statistical Physics of Protein Sequences: A Key Issues Review
• Fast pseudolikelihood maximization for direct-coupling analysis of protein structure from many homologous amino-acid sequences
• How pairwise coevolutionary models capture the collective residue variability in proteins
• Efficient generative modeling of protein sequences using simple autoregressive models
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Supervised Hebbian Learning
• Mapping of attention mechanisms to a generalized Potts model
• Learning through atypical "phase transitions" in overparameterized neural networks
• Consistent inference of a general model using the pseudo-likelihood method
• Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks
• Unsupervised hierarchical clustering using the learning dynamics of RBMs
• Tractability from overparametrization: The example of the negative perceptron
• Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
• From statistical inference to a differential learning rule for stochastic neural networks
• Fluctuations and the limit of predictability in protein evolution
• Dynamical mean field approach to associative memory model with non-monotonic transfer functions