Heuristic-Guided Reinforcement Learning

Abstract

We provide a framework for accelerating reinforcement learning (RL) algorithms by heuristics constructed from domain knowledge or offline data. Tabula rasa RL algorithms require environment interactions or computation that scales with the horizon of the sequential decision-making task. Using our framework, we show how heuristic-guided RL induces a much shorter-horizon subproblem that provably solves the original task. Our framework can be viewed as a horizon-based regularization for controlling bias and variance in RL under a finite interaction budget. On the theoretical side, we characterize properties of a good heuristic and its impact on RL acceleration. In particular, we introduce the novel concept of an improvable heuristic, a heuristic that allows an RL agent to extrapolate beyond its prior knowledge. On the empirical side, we instantiate our framework to accelerate several state-of-the-art algorithms in simulated robotic control tasks and procedurally generated games. Our framework complements the rich literature on warm-starting RL with expert demonstrations or exploratory datasets, and introduces a principled method for injecting prior knowledge into RL.

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• The FF Planning System: Fast Plan Generation Through Heuristic Search
• Dota 2 with Large Scale Deep Reinforcement Learning
• Conservative Q-Learning for Offline Reinforcement Learning
• Deep Q-learning from Demonstrations
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
• Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Is Pessimism Provably Efficient for Offline RL?
• Beyond the One Step Greedy Approach in Reinforcement Learning
• Discount Factor as a Regularizer in Reinforcement Learning
• Policy Improvement via Imitation of Multiple Oracles
• Regularized Behavior Value Estimation
• Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizons
• Separating value functions across time-scales

Cited by in corpus (3)

• SACHA: Soft Actor-Critic with Heuristic-Based Attention for Partially Observable Multi-Agent Path Finding
• Learning Logic Specifications for Policy Guidance in POMDPs: an Inductive Logic Programming Approach
• Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning