Occam's razor is insufficient to infer the preferences of irrational agents
arXiv:1712.05812
Abstract
Inverse reinforcement learning (IRL) attempts to infer human rewards or preferences from observed behavior. Since human planning systematically deviates from rationality, several approaches have been tried to account for specific human shortcomings. However, the general problem of inferring the reward function of an agent of unknown rationality has received little attention. Unlike the well-known ambiguity problems in IRL, this one is practically relevant but cannot be resolved by observing the agent's policy in enough environments. This paper shows (1) that a No Free Lunch result implies it is impossible to uniquely decompose a policy into a planning algorithm and reward function, and (2) that even with a reasonable simplicity prior/Occam's razor on the set of decompositions, we cannot distinguish between the true decomposition and others that lead to high regret. To address this, we need simple `normative' assumptions, which cannot be deduced exclusively from observations.
Cited by in corpus (13)
- Artificial Intelligence, Values and Alignment
- ROAD: The ROad event Awareness Dataset for Autonomous Driving
- Beyond Preferences in AI Alignment
- Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations
- Risks from Learned Optimization in Advanced Machine Learning Systems
- Impossibility Results in AI: A Survey
- Open Problems in Cooperative AI
- On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference
- On Controllability of AI
- Uncertain Decisions Facilitate Better Preference Learning
- Rational AI: A comparison of human and AI responses to triggers of economic irrationality in poker
- Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts
- Combining Reward Information from Multiple Sources