Actor-Critic Policy Optimization in Partially Observable Multiagent Environments

Abstract

Optimization of parameterized policies for reinforcement learning (RL) is an important and challenging problem in artificial intelligence. Among the most common approaches are algorithms based on gradient ascent of a score function representing discounted return. In this paper, we examine the role of these policy gradient and actor-critic algorithms in partially-observable multiagent environments. We show several candidate policy update rules and relate them to a foundation of regret minimization and multiagent learning techniques for the one-shot and tabular cases, leading to previously unknown convergence guarantees. We apply our method to model-free multiagent reinforcement learning in adversarial sequential decision problems (zero-sum imperfect information games), using RL-style function approximation. We evaluate on commonly used benchmark Poker domains, showing performance against fixed policies and empirical convergence to approximate Nash equilibria in self-play with rates similar to or better than a baseline model-free algorithm for zero sum games, without any domain-specific state space reductions.

arXiv commentNeurIPS 2018

Cited by in corpus (31)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• A Theoretical Analysis of Deep Q-Learning
• OpenSpiel: A Framework for Reinforcement Learning in Games
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games
• Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity
• TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game
• Deep Counterfactual Regret Minimization
• From Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization
• Single Deep Counterfactual Regret Minimization
• Difference Rewards Policy Gradients
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• Neural Replicator Dynamics
• DREAM: Deep Regret minimization with Advantage baselines and Model-free learning
• The Advantage Regret-Matching Actor-Critic
• Model-free Neural Counterfactual Regret Minimization with Bootstrap Learning
• Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium
• Finite-Sample Analysis For Decentralized Batch Multi-Agent Reinforcement Learning With Networked Agents
• Towards General Function Approximation in Zero-Sum Markov Games
• Approximate exploitability: Learning a best response in large games
• Low-Variance and Zero-Variance Baselines for Extensive-Form Games
• Pose-Assisted Multi-Camera Collaboration for Active Object Tracking
• Small Nash Equilibrium Certificates in Very Large Games
• Alternative Function Approximation Parameterizations for Solving Games: An Analysis of $f$-Regression Counterfactual Regret Minimization
• Competitive Policy Optimization
• Data Centers Job Scheduling with Deep Reinforcement Learning
• Tactical Reward Shaping: Bypassing Reinforcement Learning with Strategy-Based Goals
• Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus
• The Partially Observable History Process