Quantum Natural Policy Gradients: Towards Sample-Efficient Reinforcement Learning

Abstract

Reinforcement learning is a growing field in AI with a lot of potential. Intelligent behavior is learned automatically through trial and error in interaction with the environment. However, this learning process is often costly. Using variational quantum circuits as function approximators potentially can reduce this cost. In order to implement this, we propose the quantum natural policy gradient (QNPG) algorithm -- a second-order gradient-based routine that takes advantage of an efficient approximation of the quantum Fisher information matrix. We experimentally demonstrate that QNPG outperforms first-order based training on Contextual Bandits environments regarding convergence speed and stability and moreover reduces the sample complexity. Furthermore, we provide evidence for the practical feasibility of our approach by training on a 12-qubit hardware device.

arXiv commentAccepted to the 1st International Workshop on Quantum Machine Learning: From Foundations to Applications (QML@QCE 2023), Bellevue, Washington, USA. 6 pages, 4 figures, 1 table

References in corpus (19)

• Quantum Circuit Learning
• Parameterized quantum circuits as machine learning models
• Quantum optimization using variational algorithms on near-term quantum devices
• Data re-uploading for a universal quantum classifier
• A rigorous and robust quantum speed-up in supervised machine learning
• Quantum Natural Gradient
• New insights and perspectives on the natural gradient method
• Noise Resilience of Variational Quantum Compiling
• Evaluating the noise resilience of variational quantum algorithms
• On the Quantum versus Classical Learnability of Discrete Distributions
• On the natural gradient for variational quantum eigensolver
• A simple algorithm to find the L-curve corner in the regularisation of ill-posed inverse problems
• A Survey on Quantum Reinforcement Learning
• Uncovering Instabilities in Variational-Quantum Deep Q-Networks
• An Empirical Comparison of Optimizers for Quantum Machine Learning with SPSA-based Gradients
• Optimal training of variational quantum algorithms without barren plateaus
• Quantum Geometric Tensor (Fubini-Study Metric) in Simple Quantum System: A pedagogical Introduction
• Quantum policy gradient algorithms
• Quantum Speedup of Natural Gradient for Variational Bayes

Cited by in corpus (5)

• Quantum reinforcement learning in continuous action space
• Guided-SPSA: Simultaneous Perturbation Stochastic Approximation assisted by the Parameter Shift Rule
• Quantum natural gradient with thermal-state initialization
• Qiskit-Torch-Module: Fast Prototyping of Quantum Neural Networks
• Warm-Start Variational Quantum Policy Iteration