Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement

Abstract

Multi-task reinforcement learning (RL) aims to simultaneously learn policies for solving many tasks. Several prior works have found that relabeling past experience with different reward functions can improve sample efficiency. Relabeling methods typically ask: if, in hindsight, we assume that our experience was optimal for some task, for what task was it optimal? In this paper, we show that hindsight relabeling is inverse RL, an observation that suggests that we can use inverse RL in tandem for RL algorithms to efficiently solve many tasks. We use this idea to generalize goal-relabeling techniques from prior work to arbitrary classes of tasks. Our experiments confirm that relabeling data using inverse RL accelerates learning in general multi-task settings, including goal-reaching, domains with discrete sets of rewards, and those with linear reward functions.

References in corpus (23)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Theoretical Models of Learning to Learn
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Reinforcement Learning with Deep Energy-Based Policies
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Hindsight Experience Replay
• Optimal control as a graphical model inference problem
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Maximum a Posteriori Policy Optimisation
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Gradient Surgery for Multi-Task Learning
• Investigating Human Priors for Playing Video Games
• Semi-parametric Topological Memory for Navigation
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• Learning to Reach Goals via Iterated Supervised Learning
• Learning Latent Plans from Play
• Divide-and-Conquer Reinforcement Learning

Cited by in corpus (13)

• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Automating Reinforcement Learning with Example-based Resets
• Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Learning Multi-Task Transferable Rewards via Variational Inverse Reinforcement Learning
• A view on learning robust goal-conditioned value functions: Interplay between RL and MPC
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
• Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learning
• Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation