Training Agents using Upside-Down Reinforcement Learning

Abstract

We develop Upside-Down Reinforcement Learning (UDRL), a method for learning to act using only supervised learning techniques. Unlike traditional algorithms, UDRL does not use reward prediction or search for an optimal policy. Instead, it trains agents to follow commands such as "obtain so much total reward in so much time." Many of its general principles are outlined in a companion report; the goal of this paper is to develop a practical learning algorithm and show that this conceptually simple perspective on agent training can produce a range of rewarding behaviors for multiple episodic environments. Experiments show that on some tasks UDRL's performance can be surprisingly competitive with, and even exceed that of some traditional baseline algorithms developed over decades of research. Based on these results, we suggest that alternative approaches to expected reward maximization have an important role to play in training useful autonomous agents.

arXiv commentExtends NeurIPS 2019 Deep Reinforcement Learning workshop presentation

References in corpus (29)

• Adam: A Method for Stochastic Optimization
• Deep Learning in Neural Networks: An Overview
• The NumPy array: a structure for efficient numerical computation
• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Asynchronous Methods for Deep Reinforcement Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Tune: A Research Platform for Distributed Model Selection and Training
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Evolutionary Algorithms for Reinforcement Learning
• Deep Reinforcement Learning that Matters
• Hindsight Experience Replay
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Model-Free Episodic Control
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Self-Imitation Learning
• RUDDER: Return Decomposition for Delayed Rewards
• Deep Reinforcement Learning and the Deadly Triad
• Neural Episodic Control
• Learning Parameterized Skills
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• Hindsight policy gradients
• Learning to Reach Goals via Iterated Supervised Learning
• Learning to Reason with Third-Order Tensor Products
• Hindsight Credit Assignment
• Reward-Conditioned Policies
• Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions

Cited by in corpus (12)

• On the Opportunities and Risks of Foundation Models
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Reward-Conditioned Policies
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
• Simplifying Deep Reinforcement Learning via Self-Supervision
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Planning from Pixels using Inverse Dynamics Models
• Improved Memories Learning
• Evolutionary Stochastic Policy Distillation
• Inverse Policy Evaluation for Value-based Sequential Decision-making