Don't Forget Your Teacher: A Corrective Reinforcement Learning Framework

Abstract

Although reinforcement learning (RL) can provide reliable solutions in many settings, practitioners are often wary of the discrepancies between the RL solution and their status quo procedures. Therefore, they may be reluctant to adapt to the novel way of executing tasks proposed by RL. On the other hand, many real-world problems require relatively small adjustments from the status quo policies to achieve improved performance. Therefore, we propose a student-teacher RL mechanism in which the RL (the "student") learns to maximize its reward, subject to a constraint that bounds the difference between the RL policy and the "teacher" policy. The teacher can be another RL policy (e.g., trained under a slightly different setting), the status quo policy, or any other exogenous policy. We formulate this problem using a stochastic optimization model and solve it using a primal-dual policy gradient algorithm. We prove that the policy is asymptotically optimal. However, a naive implementation suffers from high variance and convergence to a stochastic optimal policy. With a few practical adjustments to address these issues, our numerical experiments confirm the effectiveness of our proposed method in multiple GridWorld scenarios.

References in corpus (8)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• AI Safety Gridworlds
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Programmatically Interpretable Reinforcement Learning
• Kickstarting Deep Reinforcement Learning
• A Deep Q-Network for the Beer Game: A Deep Reinforcement Learning algorithm to Solve Inventory Optimization Problems