A Distributional View on Multi-Objective Policy Optimization

Abstract

Many real-world problems require trading off multiple competing objectives. However, these objectives are often in different units and/or scales, which can make it challenging for practitioners to express numerical preferences over objectives in their native units. In this paper we propose a novel algorithm for multi-objective reinforcement learning that enables setting desired preferences for objectives in a scale-invariant way. We propose to learn an action distribution for each objective, and we use supervised learning to fit a parametric policy to a combination of these distributions. We demonstrate the effectiveness of our approach on challenging high-dimensional real and simulated robotics tasks, and show that setting different preferences in our framework allows us to trace out the space of nondominated solutions.

Cited by in corpus (7)

• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• A Practical Guide to Multi-Objective Reinforcement Learning and Planning
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
• Data-efficient visuomotor policy training using reinforcement learning and generative models
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Gradient-based algorithms for multi-objective bi-level optimization