Meta-learning within Projective Simulation

Abstract

Learning models of artificial intelligence can nowadays perform very well on a large variety of tasks. However, in practice different task environments are best handled by different learning models, rather than a single, universal, approach. Most non-trivial models thus require the adjustment of several to many learning parameters, which is often done on a case-by-case basis by an external party. Meta-learning refers to the ability of an agent to autonomously and dynamically adjust its own learning parameters, or meta-parameters. In this work we show how projective simulation, a recently developed model of artificial intelligence, can naturally be extended to account for meta-learning in reinforcement learning settings. The projective simulation approach is based on a random walk process over a network of clips. The suggested meta-learning scheme builds upon the same design and employs clip networks to monitor the agent's performance and to adjust its meta-parameters "on the fly". We distinguish between "reflexive adaptation" and "adaptation through learning", and show the utility of both approaches. In addition, a trade-off between flexibility and learning-time is addressed. The extended model is examined on three different kinds of reinforcement learning tasks, in which the agent has different optimal values of the meta-parameters, and is shown to perform well, reaching near-optimal to optimal success rates in all of them, without ever needing to manually adjust any meta-parameter.

14 pages, 12 figures

References in corpus (10)

• Theoretical Models of Learning to Learn
• Exponential algorithmic speedup by quantum walk
• Quantum speedup for active learning agents
• Bandit Problems with Side Observations
• Quantum walks can find a marked element on any graph
• Coherent controlization using superconducting qubits
• Projective simulation for classical learning agents: a comprehensive investigation
• Projective simulation with generalization
• Quantum-enhanced deliberation of learning agents using trapped ions
• Recommending Learning Algorithms and Their Associated Hyperparameters

Cited by in corpus (18)

• Quantum-enhanced machine learning
• Active learning machine learns to create new quantum experiments
• Optimizing Quantum Error Correction Codes with Reinforcement Learning
• Machine learning for long-distance quantum communication
• Quantum Machine Learning: from physics to software engineering
• Advances in Quantum Reinforcement Learning
• Projective simulation with generalization
• Machine learning \& artificial intelligence in the quantum domain
• Setting up experimental Bell test with reinforcement learning
• Speeding-up the decision making of a learning agent using an ion trap quantum processor
• Photonic architecture for reinforcement learning
• Collective defense of honeybee colonies: experimental results and theoretical modeling
• Benchmarking projective simulation in navigation problems
• Development of swarm behavior in artificial learning agents that adapt to different foraging environments
• On the convergence of projective-simulation-based reinforcement learning in Markov decision processes
• Optimal foraging strategies can be learned
• A Projective Simulation Scheme for Partially-Observable Multi-Agent Systems
• Quantum circuit synthesis of Bell and GHZ states using projective simulation in the NISQ era