Distributed Policy Evaluation Under Multiple Behavior Strategies
arXiv:1312.7606
Abstract
We apply diffusion strategies to develop a fully-distributed cooperative reinforcement learning algorithm in which agents in a network communicate only with their immediate neighbors to improve predictions about their environment. The algorithm can also be applied to off-policy learning, meaning that the agents can predict the response to a behavior different from the actual policies they are following. The proposed distributed strategy is efficient, with linear complexity in both computation time and memory footprint. We provide a mean-square-error performance analysis and establish convergence under constant step-size updates, which endow the network with continuous learning capabilities. The results show a clear gain from cooperation: when the individual agents can estimate the solution, cooperation increases stability and reduces bias and variance of the prediction error; but, more importantly, the network is able to approach the optimal solution even when none of the individual agents can (e.g., when the individual behavior policies restrict each agent to sample a small portion of the state space).
arXiv comment36 pages, 4 figures, accepted for publication on IEEE Transactions on Automatic Control
References in corpus (7)
- Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
- Diffusion Strategies Outperform Consensus Strategies for Distributed Estimation over Adaptive Networks
- Convergence Rate Analysis of Distributed Gossip (Linear Parameter) Estimation: Fundamental Limits and Tradeoffs
- -Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
- Performance Limits for Distributed Estimation Over LMS Adaptive Networks
- On the Learning Behavior of Adaptive Networks - Part I: Transient Analysis
- Asynchronous Adaptation and Learning over Networks - Part II: Performance Analysis