Meta-Reinforcement Learning of Structured Exploration Strategies

Abstract

Exploration is a fundamental challenge in reinforcement learning (RL). Many of the current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we explore how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm -- model agnostic exploration with structured noise (MAESN) -- to learn exploration strategies from prior experience. The prior experience is used both to initialize a policy and to acquire a latent exploration space that can inject structured stochasticity into a policy, producing exploration strategies that are informed by prior knowledge and are more effective than random action-space noise. We show that MAESN is more effective at learning exploration strategies when compared to prior meta-RL methods, RL without learned exploration strategies, and task-agnostic exploration methods. We evaluate our method on a variety of simulated tasks: locomotion with a wheeled robot, locomotion with a quadrupedal walker, and object manipulation.

References in corpus (14)

• Continuous control with deep reinforcement learning
• Prototypical Networks for Few-shot Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Noisy Networks for Exploration
• VIME: Variational Information Maximizing Exploration
• Deep Exploration via Bootstrapped DQN
• Learning to reinforcement learn
• Parameter Space Noise for Exploration
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• One-Shot Visual Imitation Learning via Meta-Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning

Cited by in corpus (52)

• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Learning to Generalize from Sparse and Underspecified Rewards
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Meta reinforcement learning as task inference
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning
• Unsupervised Meta-Learning for Reinforcement Learning
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Meta Reinforcement Learning for Optimal Design of Legged Robots
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• Toward Discretization-Consistent Closure Schemes for Large Eddy Simulation Using Reinforcement Learning
• AdaptNet: Policy Adaptation for Physics-Based Character Control
• Learning latent state representation for speeding up exploration
• Online Meta-Critic Learning for Off-Policy Actor-Critic Methods
• Meta-Adaptive Nonlinear Control: Theory and Algorithms
• Hypersolvers: Toward Fast Continuous-Depth Models
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Regularization Matters in Policy Optimization
• Meta-learning curiosity algorithms
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
• Concurrent Meta Reinforcement Learning
• Context Meta-Reinforcement Learning via Neuromodulation
• Meta-Learning for Contextual Bandit Exploration
• Disentangled Skill Embeddings for Reinforcement Learning
• Meta Reinforcement Learning with Task Embedding and Shared Policy
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Curriculum in Gradient-Based Meta-Reinforcement Learning
• Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning
• VFunc: a Deep Generative Model for Functions
• Deep Reinforcement Learning for Demand Driven Services in Logistics and Transportation Systems: A Survey
• Reinforcement Learning for Few-Shot Text Generation Adaptation
• A Survey of Exploration Methods in Reinforcement Learning
• Influence-Based Multi-Agent Exploration
• Meta-Learning Dynamics Forecasting Using Task Inference
• REIN-2: Giving Birth to Prepared Reinforcement Learning Agents Using Reinforcement Learning Agents
• Linear Representation Meta-Reinforcement Learning for Instant Adaptation
• Meta-Model-Based Meta-Policy Optimization
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Population-Based Evolution Optimizes a Meta-Learning Objective
• Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning
• Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization
• Meta-Learning Strategies through Value Maximization in Neural Networks
• One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
• A Meta Reinforcement Learning-based Approach for Self-Adaptive System
• Towards Enabling Meta-Learning from Target Models
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning
• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy