Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines

Abstract

Free energy-based reinforcement learning (FERL) with clamped quantum Boltzmann machines (QBM) was shown to significantly improve the learning efficiency compared to classical Q-learning with the restriction, however, to discrete state-action space environments. In this paper, the FERL approach is extended to multi-dimensional continuous state-action space environments to open the doors for a broader range of real-world applications. First, free energy-based Q-learning is studied for discrete action spaces, but continuous state spaces and the impact of experience replay on sample efficiency is assessed. In a second step, a hybrid actor-critic scheme for continuous state-action spaces is developed based on the Deep Deterministic Policy Gradient algorithm combining a classical actor network with a QBM-based critic. The results obtained with quantum annealing, both simulated and with D-Wave quantum annealing hardware, are discussed, and the performance is compared to classical reinforcement learning methods. The environments used throughout represent existing particle accelerator beam lines at the European Organisation for Nuclear Research (CERN). Among others, the hybrid actor-critic agent is evaluated on the actual electron beam line of the Advanced Plasma Wakefield Experiment (AWAKE).

arXiv comment17 pages, 15 figures, to be submitted to "Quantum" journal

References in corpus (12)

• Quantum Machine Learning
• An introduction to quantum machine learning
• Tune: A Research Platform for Distributed Model Selection and Training
• Quantum Computation by Adiabatic Evolution
• Undecidability of the Spectral Gap (short version)
• Acceleration of electrons in the plasma wakefield of a proton bunch
• Quantum speedup for active learning agents
• Prospects for Quantum Enhancement with Diabatic Quantum Annealing
• Variational Quantum Reinforcement Learning via Evolutionary Optimization
• Quantum enhancements for deep reinforcement learning in large spaces
• Free energy-based reinforcement learning using a quantum processor
• Performance of the LHC injector chain after the upgrade and potential development

Cited by in corpus (2)

• Quantum Computing for High-Energy Physics: State of the Art and Challenges. Summary of the QC4HEP Working Group
• Tensor Network for Anomaly Detection in the Latent Space of Proton Collision Events at the LHC