Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
arXiv:2002.09505
Abstract
In this paper, we introduce a novel form of value function, , that expresses the utility of transitioning from a state to a neighboring state and then acting optimally thereafter. In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that maximize this value. This formulation decouples actions from values while still learning off-policy. We highlight the benefits of this approach in terms of value function transfer, learning within redundant action spaces, and learning off-policy from state observations generated by sub-optimal or completely random policies. Code and videos are available at http://sites.google.com/view/qss-paper.
Accepted into ICML 2020
References in corpus (19)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Addressing Function Approximation Error in Actor-Critic Methods
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- Hindsight Experience Replay
- Visual Reinforcement Learning with Imagined Goals
- Deep Q-learning from Demonstrations
- Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
- Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
- Learning Plannable Representations with Causal InfoGAN
- Imitating Latent Policies from Observation
- Time-Contrastive Networks: Self-Supervised Learning from Video
- Recall Traces: Backtracking Models for Efficient Reinforcement Learning
- Provably Efficient Imitation Learning from Observation Alone
- State Alignment-based Imitation Learning
- Forward-Backward Reinforcement Learning
- Learning what you can do before doing anything
- Learning Action-Transferable Policy with Action Embedding
- Addressing Sample Complexity in Visual Tasks Using HER and Hallucinatory GANs