Hybrid Reward Architecture for Reinforcement Learning

Abstract

One of the main challenges in reinforcement learning (RL) is generalisation. In typical deep RL methods this is achieved by approximating the optimal value function with a low-dimensional representation using a deep network. While this approach works well in many domains, in domains where the optimal value function cannot easily be reduced to a low-dimensional representation, learning can be very slow and unstable. This paper contributes towards tackling such challenging domains, by proposing a new method, called Hybrid Reward Architecture (HRA). HRA takes as input a decomposed reward function and learns a separate value function for each component reward function. Because each component typically only depends on a subset of all features, the corresponding value function can be approximated more easily by a low-dimensional representation, enabling more effective learning. We demonstrate HRA on a toy-problem and the Atari game Ms. Pac-Man, where HRA achieves above-human performance.

References in corpus (2)

• Learning and Transfer of Modulated Locomotor Controllers
• Off-Policy Reward Shaping with Ensembles

Cited by in corpus (15)

• Challenges of Real-World Reinforcement Learning
• A Survey of Deep Reinforcement Learning in Video Games
• Neural Episodic Control
• Parameter Sharing Deep Deterministic Policy Gradient for Cooperative Multi-agent Reinforcement Learning
• Mutual Alignment Transfer Learning
• The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously
• A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learning
• Learning predictive representations in autonomous driving to improve deep reinforcement learning
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning
• Neural Fictitious Self-Play on ELF Mini-RTS
• Learning Independently-Obtainable Reward Functions
• Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition
• Pareto Deterministic Policy Gradients and Its Application in 5G Massive MIMO Networks
• Reinforcement Learning via Reasoning from Demonstration
• Learning robust driving policies without online exploration