Efficient Exploration via State Marginal Matching

Abstract

Exploration is critical to a reinforcement learning agent's performance in its given environment. Prior exploration methods are often based on using heuristic auxiliary predictions to guide policy behavior, lacking a mathematically-grounded objective with clear properties. In contrast, we recast exploration as a problem of State Marginal Matching (SMM), where we aim to learn a policy for which the state marginal distribution matches a given target state distribution. The target distribution is a uniform distribution in most cases, but can incorporate prior knowledge if available. In effect, SMM amortizes the cost of learning to explore in a given environment. The SMM objective can be viewed as a two-player, zero-sum game between a state density model and a parametric policy, an idea that we use to build an algorithm for optimizing the SMM objective. Using this formalism, we further demonstrate that prior work approximately maximizes the SMM objective, offering an explanation for the success of these methods. On both simulated and real-world tasks, we demonstrate that agents that directly optimize the SMM objective explore faster and adapt more quickly to new tasks as compared to prior exploration methods.

arXiv commentVideos and code: https://sites.google.com/view/state-marginal-matching

References in corpus (32)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• A Simple Neural Attentive Meta-Learner
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Noisy Networks for Exploration
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Parameter Space Noise for Exploration
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Visual Reinforcement Learning with Imagined Goals
• Optimal control as a graphical model inference problem
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Exploration by Random Network Distillation
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Maximum a Posteriori Policy Optimisation
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Automatic Goal Generation for Reinforcement Learning Agents
• Provably Efficient Maximum Entropy Exploration
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Variational Option Discovery Algorithms
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Learning to Explore with Meta-Policy Gradient
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• A Counter-Example to Karlin's Strong Conjecture for Fictitious Play
• Concurrent Meta Reinforcement Learning
• ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots
• Coordinated Exploration in Concurrent Reinforcement Learning

Cited by in corpus (45)

• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• A survey on intrinsic motivation in reinforcement learning
• Reinforcement Learning through Active Inference
• Unsupervised Meta-Learning for Reinforcement Learning
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Reinforcement Learning with Prototypical Representations
• Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
• If MaxEnt RL is the Answer, What is the Question?
• A Divergence Minimization Perspective on Imitation Learning Methods
• Action and Perception as Divergence Minimization
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• State Alignment-based Imitation Learning
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Behavior From the Void: Unsupervised Active Pre-Training
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
• Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint
• Geometric Entropic Exploration
• Reinforcement Learning by Guided Safe Exploration
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Generalized Decision Transformer for Offline Hindsight Information Matching
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning
• Energy-based Surprise Minimization for Multi-Agent Value Factorization
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching
• The Information Geometry of Unsupervised Reinforcement Learning
• Learning Compositional Neural Programs for Continuous Control
• Learning Altruistic Behaviours in Reinforcement Learning without External Rewards
• Reward is enough for convex MDPs
• Planning under Uncertainty to Goal Distributions
• Learning without Knowing: Unobserved Context in Continuous Transfer Reinforcement Learning
• Imitation with Neural Density Models
• More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences
• Joint Perception and Control as Inference with an Object-based Implementation
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching