Learning to Represent Action Values as a Hypergraph on the Action Vertices

Abstract

Action-value estimation is a critical component of many reinforcement learning (RL) methods whereby sample complexity relies heavily on how fast a good estimator for action value can be learned. By viewing this problem through the lens of representation learning, good representations of both state and action can facilitate action-value estimation. While advances in deep learning have seamlessly driven progress in learning state representations, given the specificity of the notion of agency to RL, little attention has been paid to learning action representations. We conjecture that leveraging the combinatorial structure of multi-dimensional action spaces is a key ingredient for learning good representations of action. To test this, we set forth the action hypergraph networks framework -- a class of functions for learning action representations in multi-dimensional discrete action spaces with a structural inductive bias. Using this framework we realise an agent class based on a combination with deep Q-networks, which we dub hypergraph Q-networks. We show the effectiveness of our approach on a myriad of domains: illustrative prediction problems under minimal confounding effects, Atari 2600 games, and discretised physical control benchmarks.

ICLR 2021, code: https://github.com/atavakol/action-hypergraph-networks

References in corpus (14)

• Noisy Networks for Exploration
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity
• Q-Learning in enormous action spaces via amortized approximate maximization
• One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
• Distributional Policy Optimization: An Alternative Approach for Continuous Control
• Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learning
• Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking
• Temporally-Extended ε-Greedy Exploration