Environment Probing Interaction Policies

Abstract

A key challenge in reinforcement learning (RL) is environment generalization: a policy trained to solve a task in one environment often fails to solve the same task in a slightly different test environment. A common approach to improve inter-environment transfer is to learn policies that are invariant to the distribution of testing environments. However, we argue that instead of being invariant, the policy should identify the specific nuances of an environment and exploit them to achieve better performance. In this work, we propose the 'Environment-Probing' Interaction (EPI) policy, a policy that probes a new environment to extract an implicit understanding of that environment's behavior. Once this environment-specific information is obtained, it is used as an additional input to a task-specific policy that can now perform environment-conditioned actions to solve a task. To learn these EPI-policies, we present a reward function based on transition predictability. Specifically, a higher reward is given if the trajectory generated by the EPI-policy can be used to better predict transitions. We experimentally show that EPI-conditioned task-specific policies significantly outperform commonly used policy generalization methods on novel testing environments.

arXiv commentPublished as a conference paper at ICLR 2019

Cited by in corpus (14)

• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• Causal Induction from Visual Observations for Goal Directed Tasks
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions
• RMA: Rapid Motor Adaptation for Legged Robots
• Estimating Mass Distribution of Articulated Objects using Non-prehensile Manipulation
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap
• Block Contextual MDPs for Continual Learning
• CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
• AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions