Why Neurons Have Thousands of Synapses, A Theory of Sequence Memory in Neocortex

Abstract

Neocortical neurons have thousands of excitatory synapses. It is a mystery how neurons integrate the input from so many synapses and what kind of large-scale network behavior this enables. It has been previously proposed that non-linear properties of dendrites enable neurons to recognize multiple patterns. In this paper we extend this idea by showing that a neuron with several thousand synapses arranged along active dendrites can learn to accurately and robustly recognize hundreds of unique patterns of cellular activity, even in the presence of large amounts of noise and pattern variation. We then propose a neuron model where some of the patterns recognized by a neuron lead to action potentials and define the classic receptive field of the neuron, whereas the majority of the patterns recognized by a neuron act as predictions by slightly depolarizing the neuron without immediately generating an action potential. We then present a network model based on neurons with these properties and show that the network learns a robust model of time-based sequences. Given the similarity of excitatory neurons throughout the neocortex and the importance of sequence memory in inference and behavior, we propose that this form of sequence memory is a universal property of neocortical tissue. We further propose that cellular layers in the neocortex implement variations of the same sequence memory algorithm to achieve different aspects of inference and behavior. The neuron and network models we introduce are robust over a wide range of parameters as long as the network uses a sparse distributed code of cellular activations. The sequence capacity of the network scales linearly with the number of synapses on each neuron. Thus neurons need thousands of synapses to learn the many temporal patterns in sensory stimuli and motor sequences.

Submitted for publication

References in corpus (3)

• Continuous online sequence learning with an unsupervised neural network model
• Properties of Sparse Distributed Representations and their Application to Hierarchical Temporal Memory
• Porting HTM Models to the Heidelberg Neuromorphic Computing Platform

Cited by in corpus (43)

• Neuro-memristive Circuits for Edge Computing: A review
• Continuous online sequence learning with an unsupervised neural network model
• Superconducting optoelectronic circuits for neuromorphic computing
• Real-Time Anomaly Detection for Streaming Analytics
• Towards an integration of deep learning and neuroscience
• Drawing Inspiration from Biological Dendrites to Empower Artificial Neural Networks
• Unsupervised Anomaly Detection in Stream Data with Online Evolving Spiking Neural Networks
• A Polynomial-Time Algorithm for Solving the Minimal Observability Problem in Conjunctive Boolean Networks
• Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
• Circuit designs for superconducting optoelectronic loop neurons
• Encoding Data for HTM Systems
• Neuroscience-Inspired Algorithms for the Predictive Maintenance of Manufacturing Systems
• Neuromorphic Architecture for the Hierarchical Temporal Memory
• Sequence learning, prediction, and replay in networks of spiking neurons
• End-to-End Memristive HTM System for Pattern Recognition and Sequence Prediction
• Neuromemrisitive Architecture of HTM with On-Device Learning and Neurogenesis
• An active dendritic tree can mitigate fan-in limitations in superconducting neurons
• Application of Hierarchical Temporal Memory Theory for Document Categorization
• ToyArchitecture: Unsupervised Learning of Interpretable Models of the World
• New Ideas for Brain Modelling 7
• Superconducting Optoelectronic Neurons I: General Principles
• An Accelerated Analog Neuromorphic Hardware System Emulating NMDA- and Calcium-Based Non-Linear Dendrites
• Attention Approximates Sparse Distributed Memory
• Continual Learning with Deep Artificial Neurons
• Superconducting Optoelectronic Neurons II: Receiver Circuits
• Coherent noise enables probabilistic sequence replay in spiking neuronal networks
• Superconducting Optoelectronic Neurons V: Networks and Scaling
• CLAASIC: a Cortex-Inspired Hardware Accelerator
• Feynman Machine: The Universal Dynamical Systems Computer
• New Ideas for Brain Modelling 4
• Symphony from Synapses: Neocortex as a Universal Dynamical Systems Modeller using Hierarchical Temporal Memory
• A Hippocampus Model for Online One-Shot Storage of Pattern Sequences
• New Ideas for Brain Modelling 2
• An Operating Principle of the Cerebral Cortex, and a Cellular Mechanism for Attentional Trial-and-Error Pattern Learning and Useful Classification Extraction
• Phenomenological modeling of diverse and heterogeneous synaptic dynamics at natural density
• New Ideas for Brain Modelling 5
• Associative Knowledge Graphs for Efficient Sequence Storage and Retrieval
• Resonances induced by Spiking Time Dependent Plasticity
• Toward Human-Level Artificial Intelligence
• Towards Machine Intelligence
• Formation of cell assemblies with iterative winners-take-all computation and excitation-inhibition balance
• The Foundations of Deep Learning with a Path Towards General Intelligence
• Enhancing Biologically Inspired Hierarchical Temporal Memory with Hardware-Accelerated Reflex Memory