Blockchain-assisted Demonstration Cloning for Multi-Agent Deep Reinforcement Learning

Abstract

Multi-Agent Deep Reinforcement Learning (MDRL) is a promising research area in which agents learn complex behaviors in cooperative or competitive environments. However, MDRL comes with several challenges that hinder its usability, including sample efficiency, curse of dimensionality, and environment exploration. Recent works proposing Federated Reinforcement Learning (FRL) to tackle these issues suffer from problems related to model restrictions and maliciousness. Other proposals using reward shaping require considerable engineering and could lead to local optima. In this paper, we propose a novel Blockchain-assisted Multi-Expert Demonstration Cloning (MEDC) framework for MDRL. The proposed method utilizes expert demonstrations in guiding the learning of new MDRL agents, by suggesting exploration actions in the environment. A model sharing framework on Blockchain is designed to allow users to share their trained models, which can be allocated as expert models to requesting users to aid in training MDRL systems. A Consortium Blockchain is adopted to enable traceable and autonomous execution without the need for a single trusted entity. Smart Contracts are designed to manage users and models allocation, which are shared using IPFS. The proposed framework is tested on several applications, and is benchmarked against existing methods in FRL, Reward Shaping, and Imitation Learning-assisted RL. The results show the outperformance of the proposed framework in terms of learning speed and resiliency to faulty and malicious models.

References in corpus (13)

• Knowledge Distillation: A Survey
• Federated Learning for Internet of Things: A Comprehensive Survey
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• Roulette-wheel selection via stochastic acceptance
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning
• Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• PRIMAL2: Pathfinding via Reinforcement and Imitation Multi-Agent Learning -- Lifelong
• Federated Multi-Agent Actor-Critic Learning for Age Sensitive Mobile Edge Computing
• Influence- and Interest-based Worker Recruitment in Crowdsourcing using Online Social Networks