Controlled Online Optimization Learning (COOL): Finding the ground state of spin Hamiltonians with reinforcement learning

Abstract

Reinforcement learning (RL) has become a proven method for optimizing a procedure for which success has been defined, but the specific actions needed to achieve it have not. We apply the so-called "black box" method of RL to what has been referred as the "black art" of simulated annealing (SA), demonstrating that an RL agent based on proximal policy optimization can, through experience alone, arrive at a temperature schedule that surpasses the performance of standard heuristic temperature schedules for two classes of Hamiltonians. When the system is initialized at a cool temperature, the RL agent learns to heat the system to "melt" it, and then slowly cool it in an effort to anneal to the ground state; if the system is initialized at a high temperature, the algorithm immediately cools the system. We investigate the performance of our RL-driven SA agent in generalizing to all Hamiltonians of a specific class; when trained on random Hamiltonians of nearest-neighbour spin glasses, the RL agent is able to control the SA process for other Hamiltonians, reaching the ground state with a higher probability than a simple linear annealing schedule. Furthermore, the scaling performance (with respect to system size) of the RL approach is far more favourable, achieving a performance improvement of one order of magnitude on L=14x14 systems. We demonstrate the robustness of the RL approach when the system operates in a "destructive observation" mode, an allusion to a quantum system where measurements destroy the state of the system. The success of the RL agent could have far-reaching impact, from classical optimization, to quantum annealing, to the simulation of physical systems.

References in corpus (1)

• Hierarchical Reinforcement Learning for Multi-agent MOBA Game

Cited by in corpus (12)

• Combinatorial Optimization with Physics-Inspired Graph Neural Networks
• Machine Learning for Quantum Matter
• Deep learning enhanced Rydberg multifrequency microwave recognition
• Order-of-magnitude differences in computational performance of analog Ising machines induced by the choice of nonlinearity
• Deep Learning and Crystal Plasticity: A Preconditioning Approach for Accurate Orientation Evolution Prediction
• Deep Learning of Quantum Many-Body Dynamics via Random Driving
• Hidden orders in spin-orbit entangled correlated insulators
• Supplementing Recurrent Neural Networks with Annealing to Solve Combinatorial Optimization Problems
• Message Passing Variational Autoregressive Network for Solving Intractable Ising Models
• Finding spin glass ground states through deep reinforcement learning
• Fingerprint Recognition of Partial Discharge Signals in Deep Learning Enhanced Rydberg Atomic Sensors
• Discovering quasiorder parameters in the Potts model: A bridge between machine learning and critical phenomena