Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning

Abstract

Efficient exploration remains a challenging problem in reinforcement learning, especially for tasks where extrinsic rewards from environments are sparse or even totally disregarded. Significant advances based on intrinsic motivation show promising results in simple environments but often get stuck in environments with multimodal and stochastic dynamics. In this work, we propose a variational dynamic model based on the conditional variational inference to model the multimodality and stochasticity. We consider the environmental state-action transition as a conditional generative process by generating the next-state prediction under the condition of the current state, action, and latent variable, which provides a better understanding of the dynamics and leads a better performance in exploration. We derive an upper bound of the negative log-likelihood of the environmental transition and use such an upper bound as the intrinsic reward for exploration, which allows the agent to learn skills by self-supervised exploration without observing extrinsic rewards. We evaluate the proposed method on several image-based simulation tasks and a real robotic manipulating task. Our method outperforms several state-of-the-art environment model-based exploration approaches.

IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2021

References in corpus (31)

• Auto-Encoding Variational Bayes
• Variational Inference: A Review for Statisticians
• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Noisy Networks for Exploration
• VIME: Variational Information Maximizing Exploration
• Deep Exploration via Bootstrapped DQN
• Large-Scale Study of Curiosity-Driven Learning
• Count-Based Exploration with Neural Density Models
• Hindsight Experience Replay
• Agent57: Outperforming the Atari Human Benchmark
• Episodic Curiosity through Reachability
• Randomized Prior Functions for Deep Reinforcement Learning
• Improving Generalization and Stability of Generative Adversarial Networks
• Model-Based Active Exploration
• Never Give Up: Learning Directed Exploration Strategies
• Variational Autoencoder with Arbitrary Conditioning
• Contingency-Aware Exploration in Reinforcement Learning
• Maximum Entropy-Regularized Multi-Goal Reinforcement Learning
• Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation
• Better Exploration with Optimistic Actor-Critic
• VIREL: A Variational Inference Framework for Reinforcement Learning
• On Learning Intrinsic Rewards for Policy Gradient Methods
• EMI: Exploration with Mutual Information
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Learning Novel Policies For Tasks
• MULEX: Disentangling Exploitation from Exploration in Deep RL
• Principled Exploration via Optimistic Bootstrapping and Backward Induction
• Exploration via Hindsight Goal Generation

Cited by in corpus (3)

• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Principled Exploration via Optimistic Bootstrapping and Backward Induction