My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control

Abstract

Multitask Reinforcement Learning is a promising way to obtain models with better performance, generalisation, data efficiency, and robustness. Most existing work is limited to compatible settings, where the state and action space dimensions are the same across tasks. Graph Neural Networks (GNN) are one way to address incompatible environments, because they can process graphs of arbitrary size. They also allow practitioners to inject biases encoded in the structure of the input graph. Existing work in graph-based continuous control uses the physical morphology of the agent to construct the input graph, i.e., encoding limb features as node labels and using edges to connect the nodes if their corresponded limbs are physically connected. In this work, we present a series of ablations on existing methods that show that morphological information encoded in the graph does not improve their performance. Motivated by the hypothesis that any benefits GNNs extract from the graph structure are outweighed by difficulties they create for message passing, we also propose Amorpheus, a transformer-based approach. Further results show that, while Amorpheus ignores the morphological information that GNNs encode, it nonetheless substantially outperforms GNN-based methods that use the morphological information to define the message-passing scheme.

arXiv commentICLR 2021 Camera-Ready Version

References in corpus (17)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Learning Combinatorial Optimization Algorithms over Graphs
• Layer Normalization
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Emergent Tool Use From Multi-Agent Autocurricula
• Assessing BERT's Syntactic Abilities
• Relational Deep Reinforcement Learning
• Differentiable Graph Module (DGM) for Graph Convolutional Networks
• Policy Distillation
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
• Hardware Conditioned Policies for Multi-Robot Transfer Learning
• One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
• Can $Q$-Learning with Graph Networks Learn a Generalizable Branching Heuristic for a SAT Solver?
• Learning Graph Structure With A Finite-State Automaton Layer