Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning

Abstract

Quantum machine learning (QML) has been identified as one of the key fields that could reap advantages from near-term quantum devices, next to optimization and quantum chemistry. Research in this area has focused primarily on variational quantum algorithms (VQAs), and several proposals to enhance supervised, unsupervised and reinforcement learning (RL) algorithms with VQAs have been put forward. Out of the three, RL is the least studied and it is still an open question whether VQAs can be competitive with state-of-the-art classical algorithms based on neural networks (NNs) even on simple benchmark tasks. In this work, we introduce a training method for parametrized quantum circuits (PQCs) that can be used to solve RL tasks for discrete and continuous state spaces based on the deep Q-learning algorithm. We investigate which architectural choices for quantum Q-learning agents are most important for successfully solving certain types of environments by performing ablation studies for a number of different data encoding and readout strategies. We provide insight into why the performance of a VQA-based Q-learning algorithm crucially depends on the observables of the quantum model and show how to choose suitable observables based on the learning task at hand. To compare our model against the classical DQN algorithm, we perform an extensive hyperparameter search of PQCs and NNs with varying numbers of parameters. We confirm that similar to results in classical literature, the architectural choices and hyperparameters contribute more to the agents' success in a RL setting than the number of parameters used in the model. Finally, we show when recent separation results between classical and quantum agents for policy gradient RL can be extended to inferring optimal Q-values in restricted families of environments.

arXiv commentVersion accepted in Quantum journal, 16+3 pages, 14 figures

References in corpus (37)

• Quantum Computing in the NISQ era and beyond
• Hardware-efficient Variational Quantum Eigensolver for Small Molecules and Quantum Magnets
• Supervised learning with quantum enhanced feature spaces
• Barren plateaus in quantum neural network training landscapes
• Neural Architecture Search: A Survey
• Noisy intermediate-scale quantum (NISQ) algorithms
• Quantum machine learning in feature Hilbert spaces
• Asynchronous Methods for Deep Reinforcement Learning
• Quantum Circuit Learning
• Activation Functions: Comparison of trends in Practice and Research for Deep Learning
• Parameterized quantum circuits as machine learning models
• Cost Function Dependent Barren Plateaus in Shallow Parametrized Quantum Circuits
• Circuit-centric quantum classifiers
• Dota 2 with Large Scale Deep Reinforcement Learning
• Expressibility and entangling capability of parameterized quantum circuits for hybrid quantum-classical algorithms
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Noise-Induced Barren Plateaus in Variational Quantum Algorithms
• The effect of data encoding on the expressive power of variational quantum machine learning models
• Data re-uploading for a universal quantum classifier
• OpenAI Gym
• Quantum generative adversarial learning
• A rigorous and robust quantum speed-up in supervised machine learning
• Quantum Generative Adversarial Networks for Learning and Loading Random Distributions
• Quantum Boltzmann Machine
• Layerwise learning for quantum neural networks
• Differentiable Learning of Quantum Circuit Born Machine
• Exploring entanglement and optimization within the Hamiltonian Variational Ansatz
• The Born Supremacy: Quantum Advantage and Training of an Ising Born Machine
• Variational Quantum Boltzmann Machines
• Layer VQE: A Variational Approach for Combinatorial Optimization on Noisy Quantum Computers
• Advances in Quantum Reinforcement Learning
• Adaptive pruning-based optimization of parameterized quantum circuits
• Quantum Wasserstein Generative Adversarial Networks
• Quantum enhancements for deep reinforcement learning in large spaces
• Learning Unitaries by Gradient Descent
• Quantum-accessible reinforcement learning beyond strictly epochal environments
• Parametrized quantum policies for reinforcement learning

Cited by in corpus (41)

• Challenges and Opportunities in Quantum Machine Learning
• Quantum machine learning beyond kernel methods
• Quantum Computing for High-Energy Physics: State of the Art and Challenges. Summary of the QC4HEP Working Group
• Challenges and Opportunities in Quantum Optimization
• Variational Quantum Reinforcement Learning via Evolutionary Optimization
• Theoretical Guarantees for Permutation-Equivariant Quantum Neural Networks
• Shadows of quantum machine learning
• Quantum algorithms applied to satellite mission planning for Earth observation
• Hybrid quantum ResNet for car classification and its hyperparameter optimization
• Quantum Architecture Search via Deep Reinforcement Learning
• Quantum Deep Reinforcement Learning for Robot Navigation Tasks
• Quantum Reinforcement Learning via Policy Iteration
• Uncovering Instabilities in Variational-Quantum Deep Q-Networks
• QADQN: Quantum Attention Deep Q-Network for Financial Market Prediction
• Evolutionary Quantum Architecture Search for Parametrized Quantum Circuits
• Let Quantum Neural Networks Choose Their Own Frequencies
• Quantum reinforcement learning in continuous action space
• A hybrid classical-quantum approach to speed-up Q-learning
• Near-Optimal Quantum Algorithms for Multivariate Mean Estimation
• Quantum Logic Gate Synthesis as a Markov Decision Process
• Playing Atari with Hybrid Quantum-Classical Reinforcement Learning
• VQC-Based Reinforcement Learning with Data Re-uploading: Performance and Trainability
• Neural quantum kernels: training quantum kernels with quantum neural networks
• Hyperparameter Importance of Quantum Neural Networks Across Small Datasets
• Forecasting steam mass flow in power plants using the parallel hybrid network
• Guided-SPSA: Simultaneous Perturbation Stochastic Approximation assisted by the Parameter Shift Rule
• QTN-VQC: An End-to-End Learning framework for Quantum Neural Networks
• Variational Quantum Circuit Design for Quantum Reinforcement Learning on Continuous Environments
• Hype or Heuristic? Quantum Reinforcement Learning for Join Order Optimisation
• Quantum algorithms for multivariate Monte Carlo estimation
• Framework for Learning and Control in the Classical and Quantum Domains
• Digital Quantum Simulation and Circuit Learning for the Generation of Coherent States
• Model-based Offline Quantum Reinforcement Learning
• Quantum framework for Reinforcement Learning: Integrating Markov decision process, quantum arithmetic, and trajectory search
• BCQQ: Batch-Constraint Quantum Q-Learning with Cyclic Data Re-uploading
• Learning Fourier series with parametrized quantum circuits
• Transfer Learning Analysis of Variational Quantum Circuits
• Warm-Start Variational Quantum Policy Iteration
• Expressiveness of Commutative Quantum Circuits: A Probabilistic Approach
• QAISim: A Toolkit for Modeling and Simulation of AI in Quantum Cloud Computing Environments
• Trainability issues in quantum policy gradients