Improving deep neural network performance through sampling

Abstract

Energy efficient sampling with probabilistic neurons or p-bits has been demonstrated in the context of Boltzmann machines and it is natural to ask if these approaches can be extended to the field of generative AI where energy costs have become prohibitively large. However, this very active field is dominated by feedforward deep neural networks (DNNs) which primarily use multi-bit deterministic neurons with no role for sampling. In this paper we first show that it is feasible to obtain superior accuracy through the use of multiple samples generated by probabilistic networks. This possibility raises the question of which option is energetically preferable for improving accuracy: generating more samples, or adding more bits to a single deterministic sample. We provide a simple expression that can be used to estimate these energy tradeoffs and illustrate it with results for different algorithms and architectures.

14 pages, 11 figures

References in corpus (7)

• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Weighted p-bits for FPGA implementation of probabilistic circuits
• Autonomous Probabilistic Coprocessing with Petaflips per Second
• Experimental demonstration of an integrated on-chip p-bit core utilizing stochastic Magnetic Tunnel Junctions and 2D-MoS2 FETs
• Accelerated Quantum Monte Carlo with Probabilistic Computers
• Binary stochasticity enabled highly efficient neuromorphic deep learning achieves better-than-software accuracy