Statistical Mechanics of Support Vector Networks

Abstract

Using methods of Statistical Physics, we investigate the generalization performance of support vector machines (SVMs), which have been recently introduced as a general alternative to neural networks. For nonlinear classification rules, the generalization error saturates on a plateau, when the number of examples is too small to properly estimate the coefficients of the nonlinear part. When trained on simple rules, we find that SVMs overfit only weakly. The performance of SVMs is strongly enhanced, when the distribution of the inputs has a gap in feature space.

REVTeX, 4 pages, 2 figures, accepted by Phys. Rev. Lett (typos corrected)

Cited by in corpus (33)

• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Statistical mechanics of complex neural systems and high dimensional data
• Information theoretic approach to interactive learning
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• How glassy are neural networks?
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Estimating the degree of non-Markovianity using machine learning
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Genomics models in radiotherapy: from mechanistic to machine learning
• Geometric compression of invariant manifolds in neural nets
• Numerical evidence against advantage with quantum fidelity kernels on classical data
• Asymptotic errors for convex penalized linear regression beyond Gaussian matrices
• Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation
• Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
• Random features and polynomial rules
• Analogue neural networks on correlated random graphs
• Generalization properties of finite size polynomial Support Vector Machines
• Bayesian Field Theory: Nonparametric Approaches to Density Estimation, Regression, Classification, and Inverse Quantum Problems
• The Copycat Perceptron: Smashing Barriers Through Collective Learning
• The Cavity Approach to Noisy Learning in Nonlinear Perceptrons
• On the proliferation of support vectors in high dimensions
• Error Scaling Laws for Kernel Classification under Source and Capacity Conditions
• Support vector machines and linear regression coincide with very high-dimensional features
• New Equivalences Between Interpolation and SVMs: Kernels and Structured Features
• High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve
• Soft-margin classification of object manifolds
• Statistical Mechanics of Soft Margin Classifiers
• Out-of-Distribution Generalization in Kernel Regression
• Hierarchical learning in polynomial Support Vector Machines
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• How isotropic kernels perform on simple invariants