Explaining Learned Reward Functions with Counterfactual Trajectories

Jan Wehner, Frans Oliehoek, Luciano Cavalcante Siebert

Learning rewards from human behaviour or feedback is a promising approach to aligning AI systems with human values but fails to consistently extract correct reward functions. Inter…