Interpretable Multi-Objective Reinforcement Learning through Policy Orchestration

Abstract

Autonomous cyber-physical agents and systems play an increasingly large role in our lives. To ensure that agents behave in ways aligned with the values of the societies in which they operate, we must develop techniques that allow these agents to not only maximize their reward in an environment, but also to learn and follow the implicit constraints of society. These constraints and norms can come from any number of sources including regulations, business process guidelines, laws, ethical principles, social norms, and moral values. We detail a novel approach that uses inverse reinforcement learning to learn a set of unspecified constraints from demonstrations of the task, and reinforcement learning to learn to maximize the environment rewards. More precisely, we assume that an agent can observe traces of behavior of members of the society but has no access to the explicit set of constraints that give rise to the observed behavior. Inverse reinforcement learning is used to learn such constraints, that are then combined with a possibly orthogonal value function through the use of a contextual bandit-based orchestrator that picks a contextually-appropriate choice between the two policies (constraint-based and environment reward-based) when taking actions. The contextual bandit orchestrator allows the agent to mix policies in novel ways, taking the best actions from either a reward maximizing or constrained policy. In addition, the orchestrator is transparent on which policy is being employed at each time step. We test our algorithms using a Pac-Man domain and show that the agent is able to learn to act optimally, act within the demonstrated constraints, and mix these two functions in complex ways.

arXiv comment8 pages, 3 figures

References in corpus (4)

• The social dilemma of autonomous vehicles
• Thompson Sampling for Contextual Bandits with Linear Payoffs
• AI Safety Gridworlds
• Programmatically Interpretable Reinforcement Learning

Cited by in corpus (9)

• A Survey on Practical Applications of Multi-Armed and Contextual Bandits
• Towards Interpretable-AI Policies Induction using Evolutionary Nonlinear Decision Trees for Discrete Action Systems
• A Practical Guide to Multi-Objective Reinforcement Learning and Planning
• Online learning with Corrupted context: Corrupted Contextual Bandits
• Constraint-Guided Reinforcement Learning: Augmenting the Agent-Environment-Interaction
• Contextual Bandit with Missing Rewards
• Spectral Clustering using Eigenspectrum Shape Based Nystrom Sampling
• Computing the Dirichlet-Multinomial Log-Likelihood Function
• Etat de l'art sur l'application des bandits multi-bras