Approximation with Random Bases: Pro et Contra

Abstract

In this work we discuss the problem of selecting suitable approximators from families of parameterized elementary functions that are known to be dense in a Hilbert space of functions. We consider and analyze published procedures, both randomized and deterministic, for selecting elements from these families that have been shown to ensure the rate of convergence in $L_2$ norm of order $O(1/N)$, where $N$ is the number of elements. We show that both randomized and deterministic procedures are successful if additional information about the families of functions to be approximated is provided. In the absence of such additional information one may observe exponential growth of the number of terms needed to approximate the function and/or extreme sensitivity of the outcome of the approximation to parameters. Implications of our analysis for applications of neural networks in modeling and control are illustrated with examples.

arXiv admin note: text overlap with arXiv:0905.0677

Cited by in corpus (35)

• Stochastic Configuration Networks: Fundamentals and Algorithms
• Black-box Adversarial Attacks with Limited Queries and Information
• Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priors
• Blessing of dimensionality: mathematical foundations of the statistical physics of data
• Correction of AI systems by linear discriminants: Probabilistic foundations
• The unreasonable effectiveness of small neural ensembles in high-dimensional brain
• Neutron Ghost Imaging
• Stochastic Separation Theorems
• Dynamic and Thermodynamic Models of Adaptation
• High--Dimensional Brain in a High-Dimensional World: Blessing of Dimensionality
• Knowledge Transfer Between Artificial Intelligence Systems
• Parsimonious Physics-Informed Random Projection Neural Networks for Initial-Value Problems of ODEs and index-1 DAEs
• High-dimensional brain. A tool for encoding and rapid learning of memories by single neurons
• One-Trial Correction of Legacy AI Systems and Stochastic Separation Theorems
• On the Inherent Dose-Reduction Potential of Classical Ghost Imaging
• Query-Efficient Black-box Adversarial Examples (superceded)
• General stochastic separation theorems with optimal bounds
• RandONet: Shallow-Networks with Random Projections for learning linear and nonlinear operators
• Why do Angular Margin Losses work well for Semi-Supervised Anomalous Sound Detection?
• One mask to rule them all: Writing arbitrary distributions of radiant exposure by scanning a single illuminated spatially-random screen
• High-dimensional separability for one- and few-shot learning
• How adversarial attacks can disrupt seemingly stable accurate classifiers
• Augmented Artificial Intelligence: a Conceptual Framework
• Improving Randomized Learning of Feedforward Neural Networks by Appropriate Generation of Random Parameters
• Blessing of dimensionality at the edge
• Ghost projection. II. Beam shaping using realistic spatially-random masks
• Local intrinsic dimensionality estimators based on concentration of measure
• How Powerful are Shallow Neural Networks with Bandlimited Random Weights?
• Deep Stochastic Configuration Networks with Universal Approximation Property
• Improving Neural Network Training in Low Dimensional Random Bases
• Symphony of high-dimensional brain
• High-Dimensional Vector Semantics
• Two Dimensional Stochastic Configuration Networks for Image Data Analytics
• Robust Stochastic Configuration Networks with Kernel Density Estimation
• Cross-Validation and Uncertainty Determination for Randomized Neural Networks with Applications to Mobile Sensors