Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithm

Abstract

Many engineering problems have multiple objectives, and the overall aim is to optimize a non-linear function of these objectives. In this paper, we formulate the problem of maximizing a non-linear concave function of multiple long-term objectives. A policy-gradient based model-free algorithm is proposed for the problem. To compute an estimate of the gradient, a biased estimator is proposed. The proposed algorithm is shown to achieve convergence to within an $\epsilon$ of the global optima after sampling $\mathcal{O}(\frac{M^4{\sigma }^2}{(1-\gamma {)}^8{\epsilon }^4})$ trajectories where $\gamma$ is the discount factor and $M$ is the number of the agents, thus achieving the same dependence on $\epsilon$ as the policy gradient algorithm for the standard reinforcement learning.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Risk-sensitive Reinforcement Learning
• Provably Efficient Maximum Entropy Exploration
• Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
• On the Global Convergence Rates of Softmax Policy Gradient Methods
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies
• Variational Policy Gradient Method for Reinforcement Learning with General Utilities
• An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods
• Global Optimality Guarantees For Policy Gradient Methods
• On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method
• An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient
• Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approach