Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions

Abstract

We transform reinforcement learning (RL) into a form of supervised learning (SL) by turning traditional RL on its head, calling this Upside Down RL (UDRL). Standard RL predicts rewards, while UDRL instead uses rewards as task-defining inputs, together with representations of time horizons and other computable functions of historic and desired future data. UDRL learns to interpret these input observations as commands, mapping them to actions (or action probabilities) through SL on past (possibly accidental) experience. UDRL generalizes to achieve high rewards or other goals, through input commands such as: get lots of reward within at most so much time! A separate paper [63] on first experiments with UDRL shows that even a pilot version of UDRL can outperform traditional baseline algorithms on certain challenging RL problems. We also also conceptually simplify an approach [60] for teaching a robot to imitate humans. First videotape humans imitating the robot's current behaviors, then let the robot learn through SL to map the videos (as input commands) to these behaviors, then let it generalize and imitate videos of humans executing previously unknown behavior. This Imitate-Imitator concept may actually explain why biological evolution has resulted in parents who imitate the babbling of their babies.

22 pages, 81 references

References in corpus (8)

• Deep Learning in Neural Networks: An Overview
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• RUDDER: Return Decomposition for Delayed Rewards
• Learning to Act by Predicting the Future
• Hindsight policy gradients
• Training Agents using Upside-Down Reinforcement Learning
• One Big Net For Everything

Cited by in corpus (9)

• Training Agents using Upside-Down Reinforcement Learning
• Reward-Conditioned Policies
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• On the Sample Complexity of Reinforcement Learning with Policy Space Generalization
• Linear Representation Meta-Reinforcement Learning for Instant Adaptation
• An Offline Deep Reinforcement Learning for Maintenance Decision-Making
• Policy Gradients Incorporating the Future
• Evolutionary Stochastic Policy Distillation
• Improved Memories Learning