A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning

Abstract

We develop a mathematical framework for solving multi-task reinforcement learning (MTRL) problems based on a type of policy gradient method. The goal in MTRL is to learn a common policy that operates effectively in different environments; these environments have similar (or overlapping) state spaces, but have different rewards and dynamics. We highlight two fundamental challenges in MTRL that are not present in its single task counterpart, and illustrate them with simple examples. We then develop a decentralized entropy-regularized policy gradient method for solving the MTRL problem, and study its finite-time convergence rate. We demonstrate the effectiveness of the proposed method using a series of numerical experiments. These experiments range from small-scale "GridWorld" problems that readily demonstrate the trade-offs involved in multi-task learning to large-scale problems, where common policies are learned to navigate an airborne drone in multiple (simulated) environments.

References in corpus (17)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic Algorithms and Applications
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Massively Parallel Methods for Deep Reinforcement Learning
• Learning to reinforcement learn
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• Reinforcement Learning in Healthcare: A Survey
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• Multi-agent Reinforcement Learning for Networked System Control
• Linear Time Average Consensus on Fixed Graphs and Implications for Decentralized Optimization and Multi-Agent Control
• Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning
• DisCoRL: Continual Reinforcement Learning via Policy Distillation
• Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning

Cited by in corpus (4)

• Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithms
• Multi-Task Federated Reinforcement Learning with Adversaries
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning