Linear Representation Meta-Reinforcement Learning for Instant Adaptation

Abstract

This paper introduces Fast Linearized Adaptive Policy (FLAP), a new meta-reinforcement learning (meta-RL) method that is able to extrapolate well to out-of-distribution tasks without the need to reuse data from training, and adapt almost instantaneously with the need of only a few samples during testing. FLAP builds upon the idea of learning a shared linear representation of the policy so that when adapting to a new task, it suffices to predict a set of linear weights. A separate adapter network is trained simultaneously with the policy such that during adaptation, we can directly use the adapter network to predict these linear weights instead of updating a meta-policy via gradient descent, such as in prior meta-RL methods like MAML, to obtain the new policy. The application of the separate feed-forward network not only speeds up the adaptation run-time significantly, but also generalizes extremely well to very different tasks that prior Meta-RL methods fail to generalize to. Experiments on standard continuous-control meta-RL benchmarks show FLAP presenting significantly stronger performance on out-of-distribution tasks with up to double the average return and up to 8X faster adaptation run-time speeds when compared to prior methods.

References in corpus (14)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Challenges of Real-World Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Composing Entropic Policies using Divergence Correction
• Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
• On the Sample Complexity of Reinforcement Learning with Policy Space Generalization

Cited by in corpus (3)

• Neural-Fly Enables Rapid Learning for Agile Flight in Strong Winds
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture