Byzantine-Resilient Decentralized TD Learning with Linear Function Approximation

Abstract

This paper considers the policy evaluation problem in a multi-agent reinforcement learning (MARL) environment over decentralized and directed networks. The focus is on decentralized temporal difference (TD) learning with linear function approximation in the presence of unreliable or even malicious agents, termed as Byzantine agents. In order to evaluate the quality of a fixed policy in a common environment, agents usually run decentralized TD($\lambda$) collaboratively. However, when some Byzantine agents behave adversarially, decentralized TD($\lambda$) is unable to learn an accurate linear approximation for the true value function. We propose a trimmed-mean based Byzantine-resilient decentralized TD($\lambda$) algorithm to perform policy evaluation in this setting. We establish the finite-time convergence rate, as well as the asymptotic learning error in the presence of Byzantine agents. Numerical experiments corroborate the robustness of the proposed algorithm.

References in corpus (17)

• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• Generalized Byzantine-tolerant SGD
• Byzantine Stochastic Gradient Descent
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
• Optimization for Reinforcement Learning: From Single Agent to Cooperative Agents
• Adversary-resilient Distributed and Decentralized Statistical Inference and Machine Learning: An Overview of Recent Advances Under the Byzantine Threat Model
• Phocas: dimensional Byzantine-resilient stochastic gradient descent
• A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation
• BRIDGE: Byzantine-resilient Decentralized Gradient Descent
• Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs
• Adaptive Temporal Difference Learning with Linear Function Approximation
• Iterative Approximate Byzantine Consensus in Arbitrary Directed Graphs - Part II: Synchronous and Asynchronous Systems
• Finite-Sample Analysis of Decentralized Temporal-Difference Learning with Linear Function Approximation

Cited by in corpus (3)

• Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup
• Compressed Federated Reinforcement Learning with a Generative Model
• Towards Resilience for Multi-Agent $QD$-Learning