Training Deep Boltzmann Networks with Sparse Ising Machines

Abstract

The slowing down of Moore's law has driven the development of unconventional computing paradigms, such as specialized Ising machines tailored to solve combinatorial optimization problems. In this paper, we show a new application domain for probabilistic bit (p-bit) based Ising machines by training deep generative AI models with them. Using sparse, asynchronous, and massively parallel Ising machines we train deep Boltzmann networks in a hybrid probabilistic-classical computing setup. We use the full MNIST and Fashion MNIST (FMNIST) dataset without any downsampling and a reduced version of CIFAR-10 dataset in hardware-aware network topologies implemented in moderately sized Field Programmable Gate Arrays (FPGA). For MNIST, our machine using only 4,264 nodes (p-bits) and about 30,000 parameters achieves the same classification accuracy (90%) as an optimized software-based restricted Boltzmann Machine (RBM) with approximately 3.25 million parameters. Similar results follow for FMNIST and CIFAR-10. Additionally, the sparse deep Boltzmann network can generate new handwritten digits and fashion products, a task the 3.25 million parameter RBM fails at despite achieving the same accuracy. Our hybrid computer takes a measured 50 to 64 billion probabilistic flips per second, which is at least an order of magnitude faster than superficially similar Graphics and Tensor Processing Unit (GPU/TPU) based implementations. The massively parallel architecture can comfortably perform the contrastive divergence algorithm (CD-n) with up to n = 10 million sweeps per update, beyond the capabilities of existing software implementations. These results demonstrate the potential of using Ising machines for traditionally hard-to-train deep generative Boltzmann networks, with further possible improvement in nanodevice-based realizations.

References in corpus (8)

• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• Massively Parallel Probabilistic Computing with Sparse Ising Machines
• A full-stack view of probabilistic computing with p-bits: devices, architectures and algorithms
• Carbon Emissions and Large Neural Network Training
• Demonstration of nanosecond operation in stochastic magnetic tunnel junctions
• Hardware emulation of stochastic p-bits for invertible logic
• Hardware-aware $insitu$ Boltzmann machine learning using stochastic magnetic tunnel junctions
• Training and Classification using a Restricted Boltzmann Machine on the D-Wave 2000Q

Cited by in corpus (7)

• Noise-augmented Chaotic Ising Machines for Combinatorial Optimization and Sampling
• Pushing the Boundary of Quantum Advantage in Hard Combinatorial Optimization with Probabilistic Computers
• General Oscillator-Based Ising Machine Models with Phase-Amplitude Dynamics and Polynomial Interactions
• Machine Learning Quantum Systems with Magnetic p-bits
• Many-body computing on Field Programmable Gate Arrays
• Metrics for spin-based computing
• Predicting sampling advantage of stochastic Ising Machines for Quantum Simulations