Mapping State Space using Landmarks for Universal Goal Reaching

Abstract

An agent that has well understood the environment should be able to apply its skills for any given goals, leading to the fundamental problem of learning the Universal Value Function Approximator (UVFA). A UVFA learns to predict the cumulative rewards between all state-goal pairs. However, empirically, the value function for long-range goals is always hard to estimate and may consequently result in failed policy. This has presented challenges to the learning process and the capability of neural networks. We propose a method to address this issue in large MDPs with sparse rewards, in which exploration and routing across remote states are both extremely challenging. Our method explicitly models the environment in a hierarchical manner, with a high-level dynamic landmark-based map abstracting the visited state space, and a low-level value network to derive precise local decisions. We use farthest point sampling to select landmark states from past experience, which has improved exploration compared with simple uniform sampling. Experimentally we showed that our method enables the agent to reach long-range goals at the early training stage, and achieve better performance than standard RL algorithms for a number of challenging tasks.

References in corpus (12)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Data-Efficient Hierarchical Reinforcement Learning
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Semi-parametric Topological Memory for Navigation
• Hierarchical Reinforcement Learning with Hindsight
• Composable Planning with Attributes
• Model-Based Planning with Discrete and Continuous Actions
• Robot Motion Planning in Learned Latent Spaces

Cited by in corpus (7)

• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• World Model as a Graph: Learning Latent Landmarks for Planning
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Sparse Graphical Memory for Robust Planning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Exploration via Hindsight Goal Generation
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation