Context Meta-Reinforcement Learning via Neuromodulation

Abstract

Meta-reinforcement learning (meta-RL) algorithms enable agents to adapt quickly to tasks from few samples in dynamic environments. Such a feat is achieved through dynamic representations in an agent's policy network (obtained via reasoning about task context, model parameter updates, or both). However, obtaining rich dynamic representations for fast adaptation beyond simple benchmark problems is challenging due to the burden placed on the policy network to accommodate different policies. This paper addresses the challenge by introducing neuromodulation as a modular component to augment a standard policy network that regulates neuronal activities in order to produce efficient dynamic representations for task adaptation. The proposed extension to the policy network is evaluated across multiple discrete and continuous control environments of increasing complexity. To prove the generality and benefits of the extension in meta-RL, the neuromodulated network was applied to two state-of-the-art meta-RL algorithms (CAVIA and PEARL). The result demonstrates that meta-RL augmented with neuromodulation produces significantly better result and richer dynamic representations in comparison to the baselines.

References in corpus (20)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• A Simple Neural Attentive Meta-Learner
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Similarity of Neural Network Representations Revisited
• Learning to reinforcement learn
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Fast Context Adaptation via Meta-Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Learning to Continually Learn
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Meta reinforcement learning as task inference
• Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networks
• Evolving Inborn Knowledge For Fast Adaptation in Dynamic POMDP Problems
• Exploring the Similarity of Representations in Model-Agnostic Meta-Learning

Cited by in corpus (1)

• Meta-Learning Strategies through Value Maximization in Neural Networks