Offline Meta Learning of Exploration

Abstract

Consider the following instance of the Offline Meta Reinforcement Learning (OMRL) problem: given the complete training logs of $N$ conventional RL agents, trained on $N$ different tasks, design a meta-agent that can quickly maximize reward in a new, unseen task from the same task distribution. In particular, while each conventional RL agent explored and exploited its own different task, the meta-agent must identify regularities in the data that lead to effective exploration/exploitation in the unseen task. Here, we take a Bayesian RL (BRL) view, and seek to learn a Bayes-optimal policy from the offline data. Building on the recent VariBAD BRL approach, we develop an off-policy BRL method that learns to plan an exploration strategy based on an adaptive neural belief estimate. However, learning to infer such a belief from offline data brings a new identifiability issue we term MDP ambiguity. We characterize the problem, and suggest resolutions via data collection and modification procedures. Finally, we evaluate our framework on a diverse set of domains, including difficult sparse reward tasks, and demonstrate learning of effective exploration behavior that is qualitatively different from the exploration used by any RL agent in the data.

References in corpus (23)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Reinforcement Learning with Augmented Data
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Meta reinforcement learning as task inference
• Neural Predictive Belief Representations
• Meta-learning of Sequential Strategies
• Meta-Q-Learning
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Multi-task Batch Reinforcement Learning with Metric Learning
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
• Bayesian Policy Optimization for Model Uncertainty

Cited by in corpus (7)

• Offline Meta-Reinforcement Learning with Advantage Weighting
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Knowledge is reward: Learning optimal exploration by predictive reward cashing