Discrete and Continuous Action Representation for Practical RL in Video Games

Abstract

While most current research in Reinforcement Learning (RL) focuses on improving the performance of the algorithms in controlled environments, the use of RL under constraints like those met in the video game industry is rarely studied. Operating under such constraints, we propose Hybrid SAC, an extension of the Soft Actor-Critic algorithm able to handle discrete, continuous and parameterized actions in a principled way. We show that Hybrid SAC can successfully solve a highspeed driving task in one of our games, and is competitive with the state-of-the-art on parameterized actions benchmark tasks. We also explore the impact of using normalizing flows to enrich the expressiveness of the policy at minimal computational cost, and identify a potential undesired effect of SAC when used with normalizing flows, that may be addressed by optimizing a different objective.

Presented at the AAAI-20 Workshop on Reinforcement Learning in Games

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Soft Actor-Critic for Discrete Action Settings
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space
• Latent Space Policies for Hierarchical Reinforcement Learning
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Relative Entropy Regularized Policy Iteration
• Kickstarting Deep Reinforcement Learning
• Distilling Policy Distillation
• Boosting Trust Region Policy Optimization by Normalizing Flows Policy
• Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies
• Discretizing Continuous Action Space for On-Policy Optimization
• Leveraging exploration in off-policy algorithms via normalizing flows

Cited by in corpus (8)

• Identifying optimal cycles in quantum thermal machines with reinforcement-learning
• Noise-Robust End-to-End Quantum Control using Deep Autoregressive Policy Networks
• TAAC: Temporally Abstract Actor-Critic for Continuous Control
• Investigation of Factorized Optical Flows as Mid-Level Representations
• Deep Policy Networks for NPC Behaviors that Adapt to Changing Design Parameters in Roguelike Games
• Soft Actor-Critic With Integer Actions
• Hierarchical Skills for Efficient Exploration
• AI-Based and Mobility-Aware Energy Efficient Resource Allocation and Trajectory Design for NFV Enabled Aerial Networks