GMAC: A Distributional Perspective on Actor-Critic Framework

Abstract

In this paper, we devise a distributional framework on actor-critic as a solution to distributional instability, action type restriction, and conflation between samples and statistics. We propose a new method that minimizes the Cramér distance with the multi-step Bellman target distribution generated from a novel Sample-Replacement algorithm denoted SR($\lambda$), which learns the correct value distribution under multiple Bellman operations. Parameterizing a value distribution with Gaussian Mixture Model further improves the efficiency and the performance of the method, which we name GMAC. We empirically show that GMAC captures the correct representation of value distributions and improves the performance of a conventional actor-critic method with low computational cost, in both discrete and continuous action spaces using Arcade Learning Environment (ALE) and PyBullet environment.

References in corpus (15)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Distributed Distributional Deterministic Policy Gradients
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• Fully Parameterized Quantile Function for Distributional Reinforcement Learning
• Distributional Reinforcement Learning for Efficient Exploration
• Nonlinear Distributional Gradient Temporal-Difference Learning
• Information-Directed Exploration for Deep Reinforcement Learning
• Sample-based Distributional Policy Gradient
• Distributional reinforcement learning with linear function approximation