A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

Abstract

We introduce a new algorithm for multi-objective reinforcement learning (MORL) with linear preferences, with the goal of enabling few-shot adaptation to new tasks. In MORL, the aim is to learn policies over multiple competing objectives whose relative importance (preferences) is unknown to the agent. While this alleviates dependence on scalar reward design, the expected return of a policy can change significantly with varying preferences, making it challenging to learn a single model to produce optimal policies under different preference conditions. We propose a generalized version of the Bellman equation to learn a single parametric representation for optimal policies over the space of all possible preferences. After an initial learning phase, our agent can execute the optimal policy under any given preference, or automatically infer an underlying preference with very few samples. Experiments across four different domains demonstrate the effectiveness of our approach.

arXiv commentAccepted in NeurIPS 2019

References in corpus (4)

• Continuous control with deep reinforcement learning
• Continuous Deep Q-Learning with Model-based Acceleration
• Multi-Objective Deep Reinforcement Learning
• Discrete Sequential Prediction of Continuous Actions for Deep RL

Cited by in corpus (13)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Toward Pareto Efficient Fairness-Utility Trade-off inRecommendation through Reinforcement Learning
• MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• Deep reinforcement learning for machine scheduling: Methodology, the state-of-the-art, and future directions
• A Practical Guide to Multi-Objective Reinforcement Learning and Planning
• Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• Chance-Constrained Control with Lexicographic Deep Reinforcement Learning
• Solving the scalarization issues of Advantage-based Reinforcement Learning Algorithms
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
• Multi-Agent Reinforcement Learning for Unmanned Aerial Vehicle Coordination by Multi-Critic Policy Gradient Optimization
• AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents