Learning the Pseudoinverse Solution to Network Weights

Abstract

The last decade has seen the parallel emergence in computational neuroscience and machine learning of neural network structures which spread the input signal randomly to a higher dimensional space; perform a nonlinear activation; and then solve for a regression or classification output by means of a mathematical pseudoinverse operation. In the field of neuromorphic engineering, these methods are increasingly popular for synthesizing biologically plausible neural networks, but the "learning method" - computation of the pseudoinverse by singular value decomposition - is problematic both for biological plausibility and because it is not an online or an adaptive method. We present an online or incremental method of computing the pseudoinverse, which we argue is biologically plausible as a learning method, and which can be made adaptable for non-stationary data streams. The method is significantly more memory-efficient than the conventional computation of pseudoinverses by singular value decomposition.

13 pages, 3 figures; in submission to Neural Networks

Cited by in corpus (13)

• Fast, simple and accurate handwritten digit classification by training shallow neural network classifiers with the 'extreme learning machine' algorithm
• Towards an integration of deep learning and neuroscience
• Data-Driven Fuzzy Modeling Using Deep Learning
• Linearly Convergent Randomized Iterative Methods for Computing the Pseudoinverse
• ADEPOS: Anomaly Detection based Power Saving for Predictive Maintenance using Edge Computing
• A neuromorphic hardware architecture using the Neural Engineering Framework for pattern recognition
• Storing cycles in Hopfield-type networks with pseudoinverse learning rule: admissibility and network topology
• An Online Learning Algorithm for Neuromorphic Hardware Implementation
• ADIC: Anomaly Detection Integrated Circuit in 65nm CMOS utilizing Approximate Computing
• ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks
• On Relationship of Koopman Eigenvalues and Frequencies in Dynamic Mode Decomposition
• STICK: Spike Time Interval Computational Kernel, A Framework for General Purpose Computation using Neurons, Precise Timing, Delays, and Synchrony
• A Trainable Neuromorphic Integrated Circuit that Exploits Device Mismatch