Generalized Hindsight for Reinforcement Learning

Abstract

One of the key reasons for the high sample complexity in reinforcement learning (RL) is the inability to transfer knowledge from one task to another. In standard multi-task RL settings, low-reward data collected while trying to solve one task provides little to no signal for solving that particular task and is hence effectively wasted. However, we argue that this data, which is uninformative for one task, is likely a rich source of information for other tasks. To leverage this insight and efficiently reuse data, we present Generalized Hindsight: an approximate inverse reinforcement learning technique for relabeling behaviors with the right tasks. Intuitively, given a behavior generated under one task, Generalized Hindsight returns a different task that the behavior is better suited for. Then, the behavior is relabeled with this new task before being used by an off-policy RL optimizer. Compared to standard relabeling techniques, Generalized Hindsight provides a substantially more efficient reuse of samples, which we empirically demonstrate on a suite of multi-task navigation and manipulation tasks. Videos and code can be accessed here: https://sites.google.com/view/generalized-hindsight.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Meta Learning Shared Hierarchies
• Deep Drone Racing: Learning Agile Flight in Dynamic Environments
• The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously
• Sub-policy Adaptation for Hierarchical Reinforcement Learning

Cited by in corpus (7)

• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• MHER: Model-based Hindsight Experience Replay
• Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation