Primal-Dual Distributed Temporal Difference Learning
arXiv:1805.07918
Abstract
The goal of this paper is to study a distributed version of the gradient temporal-difference (GTD) learning algorithm for a class of multi-agent Markov decision processes (MDPs). The temporal-difference (TD) learning is a reinforcement learning (RL) algorithm that learns an infinite horizon discounted cost function (or value function) for a given fixed policy without the model knowledge. In the multi-agent MDP each agent receives a local reward through a local processing. The agents communicate over sparse and random networks to learn the global value function corresponding to the aggregate of local rewards. In this paper, the problem of estimating the global value function is converted into a constrained convex optimization problem. Then, we propose a stochastic primal-dual distributed algorithm to solve it and prove that the algorithm converges to a set of solutions of the optimization problem.
arXiv commentarXiv admin note: substantial text overlap with arXiv:1803.08031
References in corpus (10)
- Constrained Consensus
- Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
- -Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
- DSA: Decentralized Double Stochastic Averaging Gradient Algorithm
- Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
- Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces
- Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
- Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning
- First-order Convergence Theory for Weakly-Convex-Weakly-Concave Min-max Problems
- Fast Multi-Agent Temporal-Difference Learning via Homotopy Stochastic Primal-Dual Optimization