Hybrid Reward Architecture for Reinforcement Learning

Abstract

One of the main challenges in reinforcement learning (RL) is generalisation. In typical deep RL methods this is achieved by approximating the optimal value function with a low-dimensional representation using a deep network. While this approach works well in many domains, in domains where the optimal value function cannot easily be reduced to a low-dimensional representation, learning can be very slow and unstable. This paper contributes towards tackling such challenging domains, by proposing a new method, called Hybrid Reward Architecture (HRA). HRA takes as input a decomposed reward function and learns a separate value function for each component reward function. Because each component typically only depends on a subset of all features, the corresponding value function can be approximated more easily by a low-dimensional representation, enabling more effective learning. We demonstrate HRA on a toy-problem and the Atari game Ms. Pac-Man, where HRA achieves above-human performance.

References in corpus (6)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Learning and Transfer of Modulated Locomotor Controllers
• Learning values across many orders of magnitude
• Learning Purposeful Behaviour in the Absence of Rewards
• Multi-Advisor Reinforcement Learning
• Off-Policy Reward Shaping with Ensembles

Cited by in corpus (42)

• Deep Reinforcement Learning: An Overview
• Challenges of Real-World Reinforcement Learning
• A Survey of Deep Reinforcement Learning in Video Games
• Acquisition of Chess Knowledge in AlphaZero
• Neural Episodic Control
• Parameter Sharing Deep Deterministic Policy Gradient for Cooperative Multi-agent Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• A Benchmark Environment Motivated by Industrial Control Problems
• Composable Planning with Attributes
• A Tutorial on Ultra-Reliable and Low-Latency Communications in 6G: Integrating Domain Knowledge into Deep Learning
• Many-Goals Reinforcement Learning
• Mutual Alignment Transfer Learning
• The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously
• Reward Design in Cooperative Multi-agent Reinforcement Learning for Packet Routing
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learning
• Contrastive Explanations for Reinforcement Learning via Embedded Self Predictions
• Learning predictive representations in autonomous driving to improve deep reinforcement learning
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning
• MoTiAC: Multi-Objective Actor-Critics for Real-Time Bidding
• Reinforcement Learning, Bit by Bit
• Neural Fictitious Self-Play on ELF Mini-RTS
• Variational Inference for Data-Efficient Model Learning in POMDPs
• Learning Independently-Obtainable Reward Functions
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
• Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimization
• Danger-aware Adaptive Composition of DRL Agents for Self-navigation
• Learning Invariances for Policy Generalization
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition
• Knowledge-Assisted Deep Reinforcement Learning in 5G Scheduler Design: From Theoretical Framework to Implementation
• Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks
• Planning in Hierarchical Reinforcement Learning: Guarantees for Using Local Policies
• Reward-Punishment Reinforcement Learning with Maximum Entropy
• Adapting to Reward Progressivity via Spectral Reinforcement Learning
• Fast Online Exact Solutions for Deterministic MDPs with Sparse Rewards
• Pareto Deterministic Policy Gradients and Its Application in 5G Massive MIMO Networks
• Reinforcement Learning via Reasoning from Demonstration
• Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learning
• Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning
• Improving On-policy Learning with Statistical Reward Accumulation
• Learning robust driving policies without online exploration