Uncovering Instabilities in Variational-Quantum Deep Q-Networks

Abstract

Deep Reinforcement Learning (RL) has considerably advanced over the past decade. At the same time, state-of-the-art RL algorithms require a large computational budget in terms of training time to converge. Recent work has started to approach this problem through the lens of quantum computing, which promises theoretical speed-ups for several traditionally hard tasks. In this work, we examine a class of hybrid quantum-classical RL algorithms that we collectively refer to as variational quantum deep Q-networks (VQ-DQN). We show that VQ-DQN approaches are subject to instabilities that cause the learned policy to diverge, study the extent to which this afflicts reproduciblity of established results based on classical simulation, and perform systematic experiments to identify potential explanations for the observed instabilities. Additionally, and in contrast to most existing work on quantum reinforcement learning, we execute RL algorithms on an actual quantum processing unit (an IBM Quantum Device) and investigate differences in behaviour between simulated and physical quantum systems that suffer from implementation deficiencies. Our experiments show that, contrary to opposite claims in the literature, it cannot be conclusively decided if known quantum approaches, even if simulated without physical imperfections, can provide an advantage as compared to classical approaches. Finally, we provide a robust, universal and well-tested implementation of VQ-DQN as a reproducible testbed for future experiments.

arXiv commentAuthors Maja Franz, Lucas Wolf, Maniraman Periyasamy contributed equally (name order randomised). To be published in the Journal of The Franklin Institute

References in corpus (18)

• Quantum Computing in the NISQ era and beyond
• Supplementary information for "Quantum supremacy using a programmable superconducting processor"
• Barren plateaus in quantum neural network training landscapes
• Quantum Circuit Learning
• End-to-End Training of Deep Visuomotor Policies
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Evaluating analytic gradients on quantum hardware
• The effect of data encoding on the expressive power of variational quantum machine learning models
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Data re-uploading for a universal quantum classifier
• Quantum reinforcement learning
• Massively Parallel Methods for Deep Reinforcement Learning
• Layerwise learning for quantum neural networks
• Agent57: Outperforming the Atari Human Benchmark
• Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
• Deep Reinforcement Learning and the Deadly Triad
• Reinforcement Learning with Quantum Variational Circuits
• Incremental Data-Uploading for Full-Quantum Classification

Cited by in corpus (9)

• Neutral Atom Quantum Computing Hardware: Performance and End-User Perspective
• Quantum Deep Reinforcement Learning for Robot Navigation Tasks
• QADQN: Quantum Attention Deep Q-Network for Financial Market Prediction
• Incremental Data-Uploading for Full-Quantum Classification
• Polynomial Reduction Methods and their Impact on QAOA Circuits
• Quantum Natural Policy Gradients: Towards Sample-Efficient Reinforcement Learning
• Hype or Heuristic? Quantum Reinforcement Learning for Join Order Optimisation
• BCQQ: Batch-Constraint Quantum Q-Learning with Cyclic Data Re-uploading
• Model-based Offline Quantum Reinforcement Learning