Estimation of Energy-dissipation Lower-bounds for Neuromorphic Learning-in-memory

Abstract

Neuromorphic or neurally-inspired optimizers rely on local but parallel parameter updates to solve problems that range from quadratic programming to Ising machines. An ideal realization of such an optimizer not only uses a compute-in-memory (CIM) paradigm to address the so-called memory-wall (i.e. energy dissipated due to repeated memory read access), but also uses a learning-in-memory (LIM) paradigm to address the energy bottlenecks due to repeated memory writes at the precision required for optimization (the update-wall), and to address the energy bottleneck due to the repeated transfer of information between short-term and long-term memories (the consolidation-wall). In this paper, we derive theoretical estimates for the energy-to-solution metric that can be achieved by this ideal neuromorphic optimizer which is realized by modulating the energy-barrier of the physical memories such that the dynamics of memory updates and memory consolidation matches the optimization or the annealing dynamics. The analysis presented in this paper captures the out-of-equilibrium thermodynamics of learning and the resulting energy-efficiency estimates are model-agnostic which only depend on the number of model-update operations (OPS), the model-size in terms of number of parameters, the speed of convergence, and the precision of the solution. To show the practical applicability of our results, we apply our analysis for estimating the lower-bound on the energy-to-solution metrics for large-scale AI workloads.

arXiv comment16 pages, 6 figures

References in corpus (12)

• Overcoming catastrophic forgetting in neural networks
• Stochastic thermodynamics: Principles and perspectives
• Nucleation mechanism for the direct graphite-to-diamond phase transition
• From a single-band metal to a high-temperature superconductor via two thermal phase transitions
• From a single-band metal to a high-temperature superconductor via two thermal phase transitions (Supporting Material)
• Top eigenvalue of a random matrix: large deviations and third order phase transition
• Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines
• Universal Bound on Energy Cost of Bit Reset in Finite Time
• Is stochastic thermodynamics the key to understanding the energy costs of computation?
• Stochastic and Quantum Thermodynamics of Driven RLC Networks
• An Adaptive Synaptic Array using Fowler-Nordheim Dynamic Analog Memory
• ON-OFF Neuromorphic ISING Machines using Fowler-Nordheim Annealers