Benchmarking projective simulation in navigation problems

Abstract

Projective simulation (PS) is a model for intelligent agents with a deliberation capacity that is based on episodic memory. The model has been shown to provide a flexible framework for constructing reinforcement-learning agents, and it allows for quantum mechanical generalization, which leads to a speed-up in deliberation time. PS agents have been applied successfully in the context of complex skill learning in robotics, and in the design of state-of-the-art quantum experiments. In this paper, we study the performance of projective simulation in two benchmarking problems in navigation, namely the grid world and the mountain car problem. The performance of PS is compared to standard tabular reinforcement learning approaches, Q-learning and SARSA. Our comparison demonstrates that the performance of PS and standard learning approaches are qualitatively and quantitatively similar, while it is much easier to choose optimal model parameters in case of projective simulation, with a reduced computational effort of one to two orders of magnitude. Our results show that the projective simulation model stands out for its simplicity in terms of the number of model parameters, which makes it simple to set up the learning agent in unknown task environments.

arXiv comment8 pages, 10 figures

References in corpus (16)

• Quantum walks: a comprehensive review
• OpenAI Gym
• Quantum-enhanced machine learning
• Learning to Navigate in Complex Environments
• Active learning machine learns to create new quantum experiments
• Quantum speedup for active learning agents
• Advances in Quantum Reinforcement Learning
• Coherent controlization using superconducting qubits
• Adaptive quantum computation in changing environments using projective simulation
• Projective simulation for classical learning agents: a comprehensive investigation
• Projective simulation with generalization
• Meta-learning within Projective Simulation
• Modelling collective motion based on the principle of agency
• Quantum-enhanced deliberation of learning agents using trapped ions
• Robotic Playing for Hierarchical Complex Skill Learning
• Skill Learning by Autonomous Robotic Playing using Active Learning and Creativity

Cited by in corpus (14)

• Optimizing Quantum Error Correction Codes with Reinforcement Learning
• Machine learning for long-distance quantum communication
• Quantum Machine Learning: from physics to software engineering
• Quantum enhancements for deep reinforcement learning in large spaces
• Setting up experimental Bell test with reinforcement learning
• Photonic architecture for reinforcement learning
• Collective defense of honeybee colonies: experimental results and theoretical modeling
• Operationally meaningful representations of physical systems in neural networks
• Development of swarm behavior in artificial learning agents that adapt to different foraging environments
• On the convergence of projective-simulation-based reinforcement learning in Markov decision processes
• Hedging using reinforcement learning: Contextual $k$-Armed Bandit versus $Q$-learning
• Quantum-accessible reinforcement learning beyond strictly epochal environments
• Performance analysis of a hybrid agent for quantum-accessible reinforcement learning
• Multi-Excitation Projective Simulation with a Many-Body Physics Inspired Inductive Bias