Simplified derivations for high-dimensional convex learning problems

Abstract

Statistical-physics calculations in machine learning and theoretical neuroscience often involve lengthy derivations that obscure physical interpretation. Here, we give concise, non-replica derivations of several key results and highlight their underlying similarities. In particular, using a cavity approach, we analyze three high-dimensional learning problems: perceptron classification of points, perceptron classification of manifolds, and kernel ridge regression. These problems share a common structure--a bipartite system of interacting feature and datum variables--enabling a unified analysis. Furthermore, for perceptron-capacity problems, we identify a symmetry that allows derivation of correct capacities through a naive method.

Submission to SciPost; 29 pages, 1 figure; revised following review by referees

References in corpus (21)

• Machine learning and the physical sciences
• Classification and Geometry of General Perceptual Manifolds
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Hopfield Networks is All You Need
• Statistical mechanics of complex neural systems and high dimensional data
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Large Associative Memory Problem in Neurobiology and Machine Learning
• Out-of-equilibrium dynamical mean-field equations for the perceptron model
• The cavity method at zero temperature
• Linear Readout of Object Manifolds
• Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
• Kernel Alignment Risk Estimator: Risk Prediction from Training Data
• Learning curves for Gaussian process regression: Approximations and bounds
• Linear Classification of Neural Manifolds with Correlated Variability
• Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
• Scaling and renormalization in high-dimensional regression
• A Dynamical Model of Neural Scaling Laws
• The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks
• Soft-margin classification of object manifolds
• Connectivity structure and dynamics of nonlinear recurrent neural networks
• Transient dynamics of associative memory models