Generative Exploration and Exploitation

Abstract

Sparse reward is one of the biggest challenges in reinforcement learning (RL). In this paper, we propose a novel method called Generative Exploration and Exploitation (GENE) to overcome sparse reward. GENE automatically generates start states to encourage the agent to explore the environment and to exploit received reward signals. GENE can adaptively tradeoff between exploration and exploitation according to the varying distributions of states experienced by the agent as the learning progresses. GENE relies on no prior knowledge about the environment and can be combined with any RL algorithm, no matter on-policy or off-policy, single-agent or multi-agent. Empirically, we demonstrate that GENE significantly outperforms existing methods in three tasks with only binary rewards, including Maze, Maze Ant, and Cooperative Navigation. Ablation studies verify the emergence of progressive exploration and automatic reversing.

arXiv commentAAAI'20

References in corpus (12)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Prioritized Experience Replay
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Large-Scale Study of Curiosity-Driven Learning
• Count-Based Exploration with Neural Density Models
• Hindsight Experience Replay
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Backplay: "Man muss immer umkehren"
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Enhanced Experience Replay Generation for Efficient Reinforcement Learning