Supervised and Unsupervised protocols for hetero-associative neural networks

Abstract

This paper introduces a learning framework for Three-Directional Associative Memory (TAM) models, extending the classical Hebbian paradigm to both supervised and unsupervised protocols within an hetero-associative setting. These neural networks consist of three interconnected layers of binary neurons interacting via generalized Hebbian synaptic couplings that allow learning, storage and retrieval of structured triplets of patterns. By relying upon glassy statistical mechanical techniques (mainly replica theory and Guerra interpolation), we analyze the emergent computational properties of these networks, at work with random (Rademacher) datasets and at the replica-symmetric level of description: we obtain a set of self-consistency equations for the order parameters that quantify the critical dataset sizes (i.e. their thresholds for learning) and describe the retrieval performance of these networks, highlighting the differences between supervised and unsupervised protocols. Numerical simulations validate our theoretical findings and demonstrate the robustness of the captured picture about TAMs also at work with structured datasets. In particular, this study provides insights into the cooperative interplay of layers, beyond that of the neurons within the layers, with potential implications for optimal design of artificial neural network architectures.

55 pages, 11 figures

References in corpus (19)

• Equilibrium statistical mechanics of bipartite spin systems
• The replica symmetric behavior of the analogical neural network
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Large Associative Memory Problem in Neurobiology and Machine Learning
• Supervised Hebbian Learning
• How glassy are neural networks?
• Neural networks with redundant representation: detecting the undetectable
• Non-Convex Multi-species Hopfield models
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Control of asymmetric Hopfield networks and application to cancer attractors
• Power-law Scaling to Assist with Key Challenges in Artificial Intelligence
• Replica symmetry breaking in dense neural networks
• Cell cycle time series gene expression data encoded as cyclic attractors in Hopfield systems
• Generalized hetero-associative neural networks
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
• The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications
• Dense Hopfield Networks in the Teacher-Student Setting
• The effect of priors on Learning with Restricted Boltzmann Machines