Warm-Start Variational Quantum Policy Iteration

Abstract

Reinforcement learning is a powerful framework aiming to determine optimal behavior in highly complex decision-making scenarios. This objective can be achieved using policy iteration, which requires to solve a typically large linear system of equations. We propose the variational quantum policy iteration (VarQPI) algorithm, realizing this step with a NISQ-compatible quantum-enhanced subroutine. Its scalability is supported by an analysis of the structure of generic reinforcement learning environments, laying the foundation for potential quantum advantage with utility-scale quantum computers. Furthermore, we introduce the warm-start initialization variant (WS-VarQPI) that significantly reduces resource overhead. The algorithm solves a large FrozenLake environment with an underlying 256x256-dimensional linear system, indicating its practical robustness.

Accepted to the IEEE International Conference on Quantum Computing and Engineering (QCE 2024), Montréal, Québec, Canada. 9 pages, 6 figures, 1 table

References in corpus (13)

• Quantum Computing in the NISQ era and beyond
• Quantum algorithm for solving linear systems of equations
• Variational Quantum Algorithms
• Noisy intermediate-scale quantum (NISQ) algorithms
• Hamiltonian Simulation by Qubitization
• Quantum algorithm for systems of linear equations with exponentially improved dependence on precision
• Quantum singular value transformation and beyond: exponential improvements for quantum matrix arithmetics
• Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
• Variational Quantum Linear Solver
• Quantum algorithms for systems of linear equations inspired by adiabatic quantum computing
• Variational algorithms for linear algebra
• Quantum Reinforcement Learning via Policy Iteration
• Quantum Natural Policy Gradients: Towards Sample-Efficient Reinforcement Learning