MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning

Abstract

This paper introduces MDP homomorphic networks for deep reinforcement learning. MDP homomorphic networks are neural networks that are equivariant under symmetries in the joint state-action space of an MDP. Current approaches to deep reinforcement learning do not usually exploit knowledge about such structure. By building this prior knowledge into policy and value networks using an equivariance constraint, we can reduce the size of the solution space. We specifically focus on group-structured symmetries (invertible transformations). Additionally, we introduce an easy method for constructing equivariant network layers numerically, so the system designer need not solve the constraints by hand, as is typically done. We construct MDP homomorphic MLPs and CNNs that are equivariant under either a group of reflections or rotations. We show that such networks converge faster than unstructured baselines on CartPole, a grid world and Pong.

References in corpus (12)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Group Equivariant Convolutional Networks
• 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• General $E(2)$-Equivariant Steerable CNNs
• 3D G-CNNs for Pulmonary Nodule Detection
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• Deep Scale-spaces: Equivariance Over Scale
• Teaching Deep Convolutional Neural Networks to Play Go
• Symmetry Learning for Function Approximation in Reinforcement Learning
• Learning to Convolve: A Generalized Weight-Tying Approach
• Goal-conditioned Batch Reinforcement Learning for Rotation Invariant Locomotion

Cited by in corpus (13)

• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Leveraging Symmetry in RL-based Legged Locomotion Control
• Contact-rich SE(3)-Equivariant Robot Manipulation Task Learning via Geometric Impedance Control
• ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning
• Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds
• Symmetry-Aware Actor-Critic for 3D Molecular Design
• E(2)-Equivariant Graph Planning for Navigation
• On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis
• Domain Adaptation of Reinforcement Learning Agents based on Network Service Proximity
• Expert-Guided Symmetry Detection in Markov Decision Processes
• Equivariant Ensembles and Regularization for Reinforcement Learning in Map-based Path Planning
• A Bayesian Approach to Invariant Deep Neural Networks