QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

Abstract

We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative examples that use the idea of factorization of the joint action-value function into individual ones for decentralized execution. VDN and QMIX address only a fraction of factorizable MARL tasks due to their structural constraint in factorization such as additivity and monotonicity. In this paper, we propose a new factorization method for MARL, QTRAN, which is free from such structural constraints and takes on a new approach to transforming the original joint action-value function into an easily factorizable one, with the same optimal actions. QTRAN guarantees more general factorization than VDN or QMIX, thus covering a much wider class of MARL tasks than does previous methods. Our experiments for the tasks of multi-domain Gaussian-squeeze and modified predator-prey demonstrate QTRAN's superior performance with especially larger margins in games whose payoffs penalize non-cooperative behavior more aggressively.

18 pages; Accepted to ICML 2019

Cited by in corpus (62)

• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Off-Policy Multi-Agent Decomposed Policy Gradients
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Succinct and Robust Multi-Agent Communication With Temporal Message Control
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Settling the Variance of Multi-Agent Policy Gradients
• Reducing Bus Bunching with Asynchronous Multi-Agent Reinforcement Learning
• Learning Individually Inferred Communication for Multi-Agent Cooperation
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• Regularized Softmax Deep Multi-Agent $Q$-Learning
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• DeepSeek-V3, GPT-4, Phi-4, and LLaMA-3.3 generate correct code for LoRaWAN-related engineering tasks
• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents
• V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL
• Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning
• A Sufficient Statistic for Influence in Structured Multiagent Environments
• SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World
• On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)
• Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning
• Multi-Agent Determinantal Q-Learning
• Context-Aware Sparse Deep Coordination Graphs
• A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning
• Value-Decomposition Multi-Agent Actor-Critics
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Learning Long-Term Reward Redistribution via Randomized Return Decomposition
• SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
• Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
• Offline Decentralized Multi-Agent Reinforcement Learning
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• Divergence-Regularized Multi-Agent Actor-Critic
• Mimicking Evolution with Reinforcement Learning
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients
• Multi-Agent Intention Sharing via Leader-Follower Forest
• Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
• AB-Mapper: Attention and BicNet Based Multi-agent Path Finding for Dynamic Crowded Environment
• Coordinated Proximal Policy Optimization
• Decentralized Circle Formation Control for Fish-like Robots in the Real-world via Reinforcement Learning
• Inducing Cooperation via Team Regret Minimization based Multi-Agent Deep Reinforcement Learning
• Calculus of Consent via MARL: Legitimating the Collaborative Governance Supplying Public Goods
• Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures
• BGC: Multi-Agent Group Belief with Graph Clustering
• Learning to Coordinate via Multiple Graph Neural Networks
• Logical Team Q-learning: An approach towards factored policies in cooperative MARL
• State-based Episodic Memory for Multi-Agent Reinforcement Learning