VQC-Based Reinforcement Learning with Data Re-uploading: Performance and Trainability
arXiv:2401.11555 · doi:10.1007/s42484-024-00190-z
Abstract
Reinforcement Learning (RL) consists of designing agents that make intelligent decisions without human supervision. When used alongside function approximators such as Neural Networks (NNs), RL is capable of solving extremely complex problems. Deep Q-Learning, a RL algorithm that uses Deep NNs, achieved super-human performance in some specific tasks. Nonetheless, it is also possible to use Variational Quantum Circuits (VQCs) as function approximators in RL algorithms. This work empirically studies the performance and trainability of such VQC-based Deep Q-Learning models in classic control benchmark environments. More specifically, we research how data re-uploading affects both these metrics. We show that the magnitude and the variance of the gradients of these models remain substantial throughout training due to the moving targets of Deep Q-Learning. Moreover, we empirically show that increasing the number of qubits does not lead to an exponential vanishing behavior of the magnitude and variance of the gradients for a PQC approximating a 2-design, unlike what was expected due to the Barren Plateau Phenomenon. This hints at the possibility of VQCs being specially adequate for being used as function approximators in such a context.
26 pages, 11 figures
References in corpus (20)
- Quantum Computing in the NISQ era and beyond
- Barren plateaus in quantum neural network training landscapes
- Quantum machine learning in feature Hilbert spaces
- Quantum Circuit Learning
- Cost Function Dependent Barren Plateaus in Shallow Parametrized Quantum Circuits
- Evaluating analytic gradients on quantum hardware
- Circuit-centric quantum classifiers
- Deep Reinforcement Learning framework for Autonomous Driving
- The effect of data encoding on the expressive power of variational quantum machine learning models
- Data re-uploading for a universal quantum classifier
- Connecting ansatz expressibility to gradient magnitudes and barren plateaus
- An initialization strategy for addressing barren plateaus in parametrized quantum circuits
- Beyond Barren Plateaus: Quantum Variational Algorithms Are Swamped With Traps
- Structure optimization for parameterized quantum circuits
- Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
- The Born Supremacy: Quantum Advantage and Training of an Ising Born Machine
- Variational Quantum Boltzmann Machines
- Theoretical Guarantees for Permutation-Equivariant Quantum Neural Networks
- Hamiltonian variational ansatz without barren plateaus
- Avoiding barren plateaus via transferability of smooth solutions in Hamiltonian Variational Ansatz