Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels

Abstract

A framework to analyze inference performance in densely connected single-layer feed-forward networks is developed for situations where a given data set is composed of correlated patterns. The framework is based on the assumption that the left and right singular value bases of the given pattern matrix are generated independently and uniformly from Haar measures. This assumption makes it possible to characterize the objective system by a single function of two variables which is determined by the eigenvalue spectrum of the cross-correlation matrix of the pattern matrix. Links to existing methods for analysis of perceptron learning and Gaussian linear vector channels and an application to a simple but nontrivial problem are also shown.

Conference paper for the International Workshop on Statistical Mechanical Informatics 2007, September 16-19, 2007, Kyoto, Japan

References in corpus (9)

• Replica Field Theory for Deterministic Models (II): A Non-Random Spin Glass with Glassy Behavior
• Learning by message-passing in networks of discrete synapses
• Mean-Field Equations for Spin Models with Orthogonal Interaction Matrices
• Analysis of CDMA systems that are characterized by eigenvalue spectrum
• Tractable approximations for probabilistic models: The adaptive TAP mean field approach
• Improved message passing for inference in densely connected systems
• The role of the interaction matrix in mean-field spin glasses
• Statistical Mechanical Development of a Sparse Bayesian Classifier
• Statistical mechanical analysis of the linear vector channel in digital communication

Cited by in corpus (28)

• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Entropy and mutual information in models of deep neural networks
• Analysis of Regularized LS Reconstruction and Random Matrix Ensembles in Compressed Sensing
• The Mutual Information in Random Linear Estimation Beyond i.i.d. Matrices
• High-temperature Expansions and Message Passing Algorithms
• Mean-field inference methods for neural networks
• Multi-Layer Generalized Linear Estimation
• Inference algorithms for gene networks: a statistical mechanics analysis
• Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)
• Perceptron capacity revisited: classification ability for correlated patterns
• The committee machine: Computational to statistical gaps in learning a two-layers neural network
• Teacher-student learning for a binary perceptron with quantum fluctuations
• Learning of correlated patterns by simple perceptrons
• A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines
• Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
• Macroscopic Analysis of Vector Approximate Message Passing in a Model Mismatch Setting
• Generalization from correlated sets of patterns in the perceptron
• Statistical mechanics of sparse generalization and model selection
• Typical $l_1$-recovery limit of sparse vectors represented by concatenations of random orthogonal matrices
• Classification and sparse-signature extraction from gene-expression data
• Partial local entropy and anisotropy in deep weight spaces
• Analysis of Bayesian Inference Algorithms by the Dynamical Functional Approach
• Blind calibration for compressed sensing: State evolution and an online algorithm
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Expectation propagation on the diluted Bayesian classifier
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Landscape Complexity for the Empirical Risk of Generalized Linear Models: Discrimination between Structured Data