PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
arXiv:2102.12560
Abstract
We study reinforcement learning (RL) with no-reward demonstrations, a setting in which an RL agent has access to additional data from the interaction of other agents with the same environment. However, it has no access to the rewards or goals of these agents, and their objectives and levels of expertise may vary widely. These assumptions are common in multi-agent settings, such as autonomous driving. To effectively use this data, we turn to the framework of successor features. This allows us to disentangle shared features and dynamics of the environment from agent-specific rewards and policies. We propose a multi-task inverse reinforcement learning (IRL) algorithm, called \emph{inverse temporal difference learning} (ITD), that learns shared state features, alongside per-agent successor features and preference vectors, purely from demonstrations without reward labels. We further show how to seamlessly integrate ITD with learning from online environment interactions, arriving at a novel algorithm for reinforcement learning with demonstrations, called -learning (pronounced `Sci-Fi'). We provide empirical evidence for the effectiveness of -learning as a method for improving RL, IRL, imitation, and few-shot transfer, and derive worst-case bounds for its performance in zero-shot transfer to new tasks.
arXiv commentThe last two authors contributed equally. Accepted at ICML 2021
References in corpus (29)
- Adam: A Method for Stochastic Optimization
- Playing Atari with Deep Reinforcement Learning
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Tune: A Research Platform for Distributed Model Selection and Training
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
- Generative Adversarial Imitation Learning
- Deep Q-learning from Demonstrations
- Maximum Entropy Deep Inverse Reinforcement Learning
- Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
- A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
- A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
- Reinforcement Learning from Imperfect Demonstrations
- Deep Successor Reinforcement Learning
- Bayesian multitask inverse reinforcement learning
- Eigenoption Discovery through the Deep Successor Representation
- Acme: A Research Framework for Distributed Reinforcement Learning
- From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
- Machine Theory of Mind
- Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?
- SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
- Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
- Observational Learning by Reinforcement Learning
- Universal Successor Features Approximators
- Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning
- Fast Task Inference with Variational Intrinsic Successor Features