Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning

Abstract

Exploration of the high-dimensional state action space is one of the biggest challenges in Reinforcement Learning (RL), especially in multi-agent domain. We present a novel technique called Experience Augmentation, which enables a time-efficient and boosted learning based on a fast, fair and thorough exploration to the environment. It can be combined with arbitrary off-policy MARL algorithms and is applicable to either homogeneous or heterogeneous environments. We demonstrate our approach by combining it with MADDPG and verifing the performance in two homogeneous and one heterogeneous environments. In the best performing scenario, the MADDPG with experience augmentation reaches to the convergence reward of vanilla MADDPG with 1/4 realistic time, and its convergence beats the original model by a significant margin. Our ablation studies show that experience augmentation is a crucial ingredient which accelerates the training process and boosts the convergence.

10 pages, 4 figures, 4 tables

References in corpus (11)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Prioritized Experience Replay
• Distributed Prioritized Experience Replay
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Mean Field Multi-Agent Reinforcement Learning
• A Deeper Look at Experience Replay
• Teaching Deep Convolutional Neural Networks to Play Go
• PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning