Reward-rational (implicit) choice: A unifying formalism for reward learning

Abstract

It is often difficult to hand-specify what the correct reward function is for a task, so researchers have instead aimed to learn reward functions from human behavior or feedback. The types of behavior interpreted as evidence of the reward function have expanded greatly in recent years. We've gone from demonstrations, to comparisons, to reading into the information leaked when the human is pushing the robot away or turning it off. And surely, there is more to come. How will a robot make sense of all these diverse types of behavior? Our key insight is that different types of behavior can be interpreted in a single unifying formalism - as a reward-rational choice that the human is making, often implicitly. The formalism offers both a unifying lens with which to view past work, as well as a recipe for interpreting new sources of information that are yet to be uncovered. We provide two examples to showcase this: interpreting a new feedback type, and reading into how the choice of feedback itself leaks information about the reward.

arXiv commentPublished at NeurIPS 2020

References in corpus (7)

• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Thermodynamics as a theory of decision-making with information processing costs
• Interactive Learning from Policy-Dependent Human Feedback
• From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
• Preferences Implicit in the State of the World
• Including Uncertainty when Learning from Human Corrections
• Active Inverse Reward Design

Cited by in corpus (12)

• Learning to summarize from human feedback
• Beyond Preferences in AI Alignment
• Learning Submodular Objectives for Team Environmental Monitoring
• Learning Latent Representations to Co-Adapt to Humans
• An Incremental Inverse Reinforcement Learning Approach for Motion Planning with Separated Path and Velocity Preferences
• PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning
• SIRL: Similarity-based Implicit Representation Learning
• Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning
• Understanding Learned Reward Functions
• The MineRL BASALT Competition on Learning from Human Feedback
• On the Effect of Robot Errors on Human Teaching Dynamics
• Learning Human Objectives from Sequences of Physical Corrections