The unreasonable effectiveness of small neural ensembles in high-dimensional brain

Abstract

Despite the widely-spread consensus on the brain complexity, sprouts of the single neuron revolution emerged in neuroscience in the 1970s. They brought many unexpected discoveries, including grandmother or concept cells and sparse coding of information in the brain. In machine learning for a long time, the famous curse of dimensionality seemed to be an unsolvable problem. Nevertheless, the idea of the blessing of dimensionality becomes gradually more and more popular. Ensembles of non-interacting or weakly interacting simple units prove to be an effective tool for solving essentially multidimensional problems. This approach is especially useful for one-shot (non-iterative) correction of errors in large legacy artificial intelligence systems. These simplicity revolutions in the era of complexity have deep fundamental reasons grounded in geometry of multidimensional data spaces. To explore and understand these reasons we revisit the background ideas of statistical physics. In the course of the 20th century they were developed into the concentration of measure theory. New stochastic separation theorems reveal the fine structure of the data clouds. We review and analyse biological, physical, and mathematical problems at the core of the fundamental question: how can high-dimensional brain organise reliable and fast learning in high-dimensional world of data by simple tools? Two critical applications are reviewed to exemplify the approach: one-shot correction of errors in intellectual systems and emergence of static and associative memories in ensembles of single neurons.

Review paper, accepted in Physics of Life Reviews; minor corrections

References in corpus (8)

• Blessing of dimensionality: mathematical foundations of the statistical physics of data
• Correction of AI systems by linear discriminants: Probabilistic foundations
• Model reduction in chemical dynamics: slow invariant manifolds, singular perturbations, thermodynamic estimates, and analysis of reaction graph
• Knowledge Transfer Between Artificial Intelligence Systems
• Hilbert's Sixth Problem: the endless road to rigour
• Data complexity measured by principal graphs
• Augmented Artificial Intelligence: a Conceptual Framework
• On the $1$-convexity of random points in the $d$-dimensional spherical layer

Cited by in corpus (10)

• Scikit-dimension: a Python package for intrinsic dimension estimation
• High--Dimensional Brain in a High-Dimensional World: Blessing of Dimensionality
• Fractional norms and quasinorms do not help to overcome the curse of dimensionality
• General stochastic separation theorems with optimal bounds
• Situation-based memory in spiking neuron-astrocyte network
• High-dimensional separability for one- and few-shot learning
• The Feasibility and Inevitability of Stealth Attacks
• Blessing of dimensionality at the edge
• Symphony of high-dimensional brain
• Linear and Fisher Separability of Random Points in the d-dimensional Spherical Layer