Evolved Policy Gradients

Abstract

We propose a metalearning approach for learning gradient-based reinforcement learning (RL) algorithms. The idea is to evolve a differentiable loss function, such that an agent, which optimizes its policy to minimize this loss, will achieve high rewards. The loss is parametrized via temporal convolutions over the agent's experience. Because this loss is highly flexible in its ability to take into account the agent's history, it enables fast task learning. Empirical results show that our evolved policy gradient algorithm (EPG) achieves faster learning on several randomized environments compared to an off-the-shelf policy gradient method. We also demonstrate that EPG's learned loss can generalize to out-of-distribution test time tasks, and exhibits qualitatively different behavior from other popular metalearning algorithms.

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• A Simple Neural Attentive Meta-Learner
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Count-Based Exploration with Neural Density Models
• Learning to learn by gradient descent by gradient descent
• One-Shot Visual Imitation Learning via Meta-Learning
• Model-Based Bayesian Exploration
• Equivalence Between Policy Gradients and Soft Q-Learning
• Learning to Optimize
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Combining policy gradient and Q-learning
• Meta-Learning Update Rules for Unsupervised Representation Learning
• UCB Exploration via Q-Ensembles
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning

Cited by in corpus (59)

• Meta-learning PINN loss functions
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Evolutionary Reinforcement Learning: A Survey
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Discovering Reinforcement Learning Algorithms
• Unsupervised Meta-Learning for Reinforcement Learning
• GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Edge-labeling Graph Neural Network for Few-shot Learning
• Efficient automatic design of robots
• AdaptNet: Policy Adaptation for Physics-Based Character Control
• Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments
• Online Meta-Critic Learning for Off-Policy Actor-Critic Methods
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Behavior Self-Organization Supports Task Inference for Continual Robot Learning
• Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems
• Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
• Meta-learning curiosity algorithms
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Using learned optimizers to make models robust to input noise
• Building Human-like Communicative Intelligence: A Grounded Perspective
• Learning Fast Adaptation with Meta Strategy Optimization
• Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Learning to learn via Self-Critique
• Multi-task Batch Reinforcement Learning with Metric Learning
• Fast Adaptation via Policy-Dynamics Value Functions
• Meta Reinforcement Learning with Task Embedding and Shared Policy
• Maximum Mutation Reinforcement Learning for Scalable Control
• GPS-Net: Graph Property Sensing Network for Scene Graph Generation
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Meta Learning Backpropagation And Improving It
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• Distributed Multi-agent Meta Learning for Trajectory Design in Wireless Drone Networks
• Learning State-Dependent Losses for Inverse Dynamics Learning
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Bottom-Up Meta-Policy Search
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Population-Based Evolution Optimizes a Meta-Learning Objective
• Importance Weighted Evolution Strategies
• Regularized Evolutionary Population-Based Training
• Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes
• Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies
• Effective Regularization Through Loss-Function Metalearning
• What is Going on Inside Recurrent Meta Reinforcement Learning Agents?
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
• Neural Auto-Curricula
• Gaussian Process Policy Optimization
• Selecting for Selection: Learning To Balance Adaptive and Diversifying Pressures in Evolutionary Search
• Meta Adaptation using Importance Weighted Demonstrations
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Policy Gradient RL Algorithms as Directed Acyclic Graphs
• On the Practical Consistency of Meta-Reinforcement Learning Algorithms
• Learning Provably Robust Motion Planners Using Funnel Libraries
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning