SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning

Abstract

A characteristic of reinforcement learning is the ability to develop unforeseen strategies when solving problems. While such strategies sometimes yield superior performance, they may also result in undesired or even dangerous behavior. In industrial scenarios, a system's behavior also needs to be predictable and lie within defined ranges. To enable the agents to learn (how) to align with a given specification, this paper proposes to explicitly transfer functional and non-functional requirements into shaped rewards. Experiments are carried out on the smart factory, a multi-agent environment modeling an industrial lot-size-one production facility, with up to eight agents and different multi-agent reinforcement learning algorithms. Results indicate that compliance with functional and non-functional constraints can be achieved by the proposed approach.

arXiv comment9 pages, 5 figures; accepted as a full paper at ICAART 2021 (http://www.icaart.org/)

References in corpus (9)

• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Counterfactual Multi-Agent Policy Gradients
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• AI Safety Gridworlds
• Emergent Coordination Through Competition
• I'm sorry Dave, I'm afraid I can't do that, Deep Q-learning from forbidden action