Relational Deep Reinforcement Learning

Abstract

We introduce an approach for deep reinforcement learning (RL) that improves upon the efficiency, generalization capacity, and interpretability of conventional approaches through structured perception and relational reasoning. It uses self-attention to iteratively reason about the relations between entities in a scene and to guide a model-free policy. Our results show that in a novel navigation and planning task called Box-World, our agent finds interpretable solutions that improve upon baselines in terms of sample complexity, ability to generalize to more complex scenes than experienced during training, and overall performance. In the StarCraft II Learning Environment, our agent achieves state-of-the-art performance on six mini-games -- surpassing human grandmaster performance on four. By considering architectural inductive biases, our work opens new directions for overcoming important, but stubborn, challenges in deep RL.

References in corpus (13)

• Semi-Supervised Classification with Graph Convolutional Networks
• StarCraft II: A New Challenge for Reinforcement Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• A Study on Overfitting in Deep Reinforcement Learning
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Non-local Neural Networks
• Towards Deep Symbolic Reinforcement Learning
• Learning model-based planning from scratch
• Visual Interaction Networks
• Relation Networks for Object Detection
• Metacontrol for Adaptive Imagination-Based Optimization
• Iterative Visual Reasoning Beyond Convolutions
• Learning to Search with MCTSnets

Cited by in corpus (60)

• Deep Reinforcement Learning for Cyber Security
• Emergent Tool Use From Multi-Agent Autocurricula
• Object-Centric Learning with Slot Attention
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• A Survey of Deep Reinforcement Learning in Video Games
• Deep Closest Point: Learning Representations for Point Cloud Registration
• Acquisition of Chess Knowledge in AlphaZero
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network
• Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
• Hierarchical Decision Making by Generating and Following Natural Language Instructions
• Graph Convolutional Reinforcement Learning
• Investigating Object Compositionality in Generative Adversarial Networks
• SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
• Graph Neural Based End-to-end Data Association Framework for Online Multiple-Object Tracking
• Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• Low-Rank Bottleneck in Multi-head Attention Models
• Graph Transformation Policy Network for Chemical Reaction Prediction
• Counterfactual Data Augmentation using Locally Factored Dynamics
• STMARL: A Spatio-Temporal Multi-Agent Reinforcement Learning Approach for Cooperative Traffic Light Control
• Relational Graph Learning for Crowd Navigation
• Incorporating Relational Background Knowledge into Reinforcement Learning via Differentiable Inductive Logic Programming
• Neuro-Symbolic Constraint Programming for Structured Prediction
• ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoning
• Learning Symbolic Rules for Interpretable Deep Reinforcement Learning
• AlignNet: Unsupervised Entity Alignment
• MEMO: A Deep Network for Flexible Combination of Episodic Memories
• On Reinforcement Learning for Full-length Game of StarCraft
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Agent-Centric Representations for Multi-Agent Reinforcement Learning
• Learning to Learn Relation for Important People Detection in Still Images
• Better Set Representations For Relational Reasoning
• Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions
• Procedural Generalization by Planning with Self-Supervised World Models
• Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems
• Deep Inductive Logic Programming meets Reinforcement Learning
• Topos and Stacks of Deep Neural Networks
• Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Tracking
• Learning to Manipulate Object Collections Using Grounded State Representations
• My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
• Deep Sets for Generalization in RL
• Control What You Can: Intrinsically Motivated Task-Planning Agent
• Semantic RL with Action Grammars: Data-Efficient Learning of Hierarchical Task Abstractions
• DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
• Detecting and Adapting to Novelty in Games
• Learning and Reasoning for Robot Dialog and Navigation Tasks
• Clear the Fog: Combat Value Assessment in Incomplete Information Games with Convolutional Encoder-Decoders
• Learning to Order Sub-questions for Complex Question Answering
• Towards Goal-Oriented Agents for Evolving Problems Observed via Conversation
• Selective Particle Attention: Visual Feature-Based Attention in Deep Reinforcement Learning
• Object-Oriented Dynamics Learning through Multi-Level Abstraction
• Deep set conditioned latent representations for action recognition
• Neural Consciousness Flow
• Attention or memory? Neurointerpretable agents in space and time
• An investigation of model-free planning
• Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment
• Interpretable UAV Collision Avoidance using Deep Reinforcement Learning
• Joint Perception and Control as Inference with an Object-based Implementation
• Attention-Based Planning with Active Perception