Toward Idealized Decision Theory

Abstract

This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of policy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research.

This is an extended version of a paper accepted to AGI-2015

Cited by in corpus (4)

• Toward negotiable reinforcement learning: shifting priorities in Pareto optimal sequential decision-making
• Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents
• Optimal Polynomial-Time Estimators: A Bayesian Notion of Approximation Algorithm
• Plausibility and probability in deductive reasoning