Q-Learning in enormous action spaces via amortized approximate maximization

Abstract

Applying Q-learning to high-dimensional or continuous action spaces can be difficult due to the required maximization over the set of possible actions. Motivated by techniques from amortized inference, we replace the expensive maximization over all actions with a maximization over a small subset of possible actions sampled from a learned proposal distribution. The resulting approach, which we dub Amortized Q-learning (AQL), is able to handle discrete, continuous, or hybrid action spaces while maintaining the benefits of Q-learning. Our experiments on continuous control tasks with up to 21 dimensional actions show that AQL outperforms D3PG (Barth-Maron et al, 2018) and QT-Opt (Kalashnikov et al, 2018). Experiments on structured discrete action spaces demonstrate that AQL can efficiently learn good policies in spaces with thousands of discrete actions.

A previous version of this work appeared at the Deep Reinforcement Learning Workshop, NeurIPS 2018

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Trust Region Policy Optimization
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Count-Based Exploration with Neural Density Models
• Continuous Deep Q-Learning with Model-based Acceleration
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Sampled Policy Gradient for Learning to Play the Game Agar.io
• Q-Networks for Binary Vector Actions

Cited by in corpus (12)

• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• A Deep Reinforcement Learning-Based Resource Scheduler for Massive MIMO Networks
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Learning Dexterous Manipulation from Suboptimal Experts
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Reversible Action Design for Combinatorial Optimization with Reinforcement Learning
• Learning to Represent Action Values as a Hypergraph on the Action Vertices
• Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization
• Dream and Search to Control: Latent Space Planning for Continuous Control
• No-Press Diplomacy from Scratch