Statistical Physics and Representations in Real and Artificial Neural Networks

Abstract

This document presents the material of two lectures on statistical physics and neural representations, delivered by one of us (R.M.) at the Fundamental Problems in Statistical Physics XIV summer school in July 2017. In a first part, we consider the neural representations of space (maps) in the hippocampus. We introduce an extension of the Hopfield model, able to store multiple spatial maps as continuous, finite-dimensional attractors. The phase diagram and dynamical properties of the model are analyzed. We then show how spatial representations can be dynamically decoded using an effective Ising model capturing the correlation structure in the neural data, and compare applications to data obtained from hippocampal multi-electrode recordings and by (sub)sampling our attractor model. In a second part, we focus on the problem of learning data representations in machine learning, in particular with artificial neural networks. We start by introducing data representations through some illustrations. We then analyze two important algorithms, Principal Component Analysis and Restricted Boltzmann Machines, with tools from statistical physics.

References in corpus (18)

• Weak pairwise correlations imply strongly correlated network states in a neural population
• Direct-coupling analysis of residue co-evolution captures native contacts across many protein families
• Understanding Deep Convolutional Networks
• A sparse coding model with synaptically local plasticity and spiking neurons can account for the diverse shapes of V1 simple cell receptive fields
• Attractor neural networks storing multiple space representations: a model for hippocampal place fields
• Inverse Ising inference using all the data
• Adaptive Cluster Expansion for Inferring Boltzmann Machines with Noisy Data
• From principal component to direct coupling analysis of coevolution in proteins: Low-eigenvalue modes are needed for structure prediction
• Small-correlation expansions for the inverse Ising problem
• Multitasking associative networks
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Adaptive cluster expansion for the inverse Ising problem: convergence, algorithm and tests
• Immune networks: multi-tasking capabilities near saturation
• High-Dimensional Inference with the generalized Hopfield Model: Principal Component Analysis and Corrections
• Transitions between spatial attractors in place-cell networks
• Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Collective motion of the activity (II)
• Cross-talk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: phase diagram (I)
• Estimating the principal components of correlation matrices from all their empirical eigenvectors

Cited by in corpus (6)

• Machine learning and the physical sciences
• Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation
• Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines
• Counting the learnable functions of structured data
• Introduction to the dynamics of disordered systems: equilibrium and gradient descent
• Dynamical prediction of two meteorological factors using the deep neural network and the long short term memory $(1)$