Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning

Abstract

Deep reinforcement learning has been successful in a variety of tasks, such as game playing and robotic manipulation. However, attempting to learn \textit{tabula rasa} disregards the logical structure of many domains as well as the wealth of readily available knowledge from domain experts that could help "warm start" the learning process. We present a novel reinforcement learning technique that allows for intelligent initialization of a neural network weights and architecture. Our approach permits the encoding domain knowledge directly into a neural decision tree, and improves upon that knowledge with policy gradient updates. We empirically validate our approach on two OpenAI Gym tasks and two modified StarCraft 2 tasks, showing that our novel architecture outperforms multilayer-perceptron and recurrent architectures. Our knowledge-based framework finds superior policies compared to imitation learning-based and prior knowledge-based approaches. Importantly, we demonstrate that our approach can be used by untrained humans to initially provide >80% increase in expected reward relative to baselines prior to training (p < 0.001), which results in a >60% increase in expected reward after policy optimization (p = 0.011).

References in corpus (14)

• Playing Atari with Deep Reinforcement Learning
• StarCraft II: A New Challenge for Reinforcement Learning
• Don't Decay the Learning Rate, Increase the Batch Size
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Deep Reinforcement Learning that Matters
• Distilling a Neural Network Into a Soft Decision Tree
• Exploration by Random Network Distillation
• Harnessing Deep Neural Networks with Logic Rules
• Interactive Learning from Policy-Dependent Human Feedback
• Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay
• Imitating Latent Policies from Observation
• Fast Policy Learning through Imitation and Reinforcement
• TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game
• Effective Warm Start for the Online Actor-Critic Reinforcement Learning based mHealth Intervention

Cited by in corpus (3)

• Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey
• Optimization Methods for Interpretable Differentiable Decision Trees in Reinforcement Learning
• Using Cognitive Models to Train Warm Start Reinforcement Learning Agents for Human-Computer Interactions