Understanding Learned Reward Functions

Abstract

In many real-world tasks, it is not possible to procedurally specify an RL agent's reward function. In such cases, a reward function must instead be learned from interacting with and observing humans. However, current techniques for reward learning may fail to produce reward functions which accurately reflect user preferences. Absent significant advances in reward learning, it is thus important to be able to audit learned reward functions to verify whether they truly capture user preferences. In this paper, we investigate techniques for interpreting learned reward functions. In particular, we apply saliency methods to identify failure modes and predict the robustness of reward functions. We find that learned reward functions often implement surprising algorithms that rely on contingent aspects of the environment. We also discover that existing interpretability techniques often attend to irrelevant changes in reward output, suggesting that reward interpretability may need significantly different methods from policy interpretability.

arXiv commentPresented at Deep RL Workshop, NeurIPS 2020

References in corpus (9)

• Sanity Checks for Saliency Maps
• Deep reinforcement learning from human preferences
• Visualizing and Understanding Atari Agents
• Causal Confusion in Imitation Learning
• Reward-rational (implicit) choice: A unifying formalism for reward learning
• Reward learning from human preferences and demonstrations in Atari
• Counterfactual Visual Explanations
• Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents
• Multi-Principal Assistance Games

Cited by in corpus (3)

• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Quantifying Differences in Reward Functions
• Inverse Decision Modeling: Learning Interpretable Representations of Behavior