Recall Traces: Backtracking Models for Efficient Reinforcement Learning

Abstract

In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the use of a backtracking model that predicts the preceding states that terminate at a given high-reward state. We can train a model which, starting from a high value state (or one that is estimated to have high value), predicts and sample for which the (state, action)-tuples may have led to that high value state. These traces of (state, action) pairs, which we refer to as Recall Traces, sampled from this backtracking model starting from a high value state, are informative as they terminate in good states, and hence we can use these traces to improve a policy. We provide a variational interpretation for this idea and a practical algorithm in which the backtracking model samples from an approximate posterior distribution over trajectories which lead to large rewards. Our method improves the sample efficiency of both on- and off-policy RL algorithms across several environments and tasks.

arXiv commentAccepted at ICLR 2019

References in corpus (17)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• End to End Learning for Self-Driving Cars
• Count-Based Exploration with Neural Density Models
• Learning Continuous Control Policies by Stochastic Value Gradients
• Continuous Deep Q-Learning with Model-based Acceleration
• Optimal control as a graphical model inference problem
• Unifying Count-Based Exploration and Intrinsic Motivation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Sample Efficient Actor-Critic with Experience Replay
• Imagination-Augmented Agents for Deep Reinforcement Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Recurrent Environment Simulators
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Combining policy gradient and Q-learning
• Forward-Backward Reinforcement Learning

Cited by in corpus (17)

• Go-Explore: a New Approach for Hard-Exploration Problems
• RUDDER: Return Decomposition for Delayed Rewards
• Imitating Latent Policies from Observation
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Generative Adversarial Self-Imitation Learning
• Backplay: "Man muss immer umkehren"
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
• Remember and Forget for Experience Replay
• Time Reversal as Self-Supervision
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
• Learning Retrospective Knowledge with Reverse Reinforcement Learning
• Competitive Experience Replay
• Learning the Arrow of Time
• Generative Exploration and Exploitation