Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization
arXiv:2102.11055
Abstract
Action-constrained reinforcement learning (RL) is a widely-used approach in various real-world applications, such as scheduling in networked systems with resource constraints and control of a robot with kinematic constraints. While the existing projection-based approaches ensure zero constraint violation, they could suffer from the zero-gradient problem due to the tight coupling of the policy gradient and the projection, which results in sample-inefficient training and slow convergence. To tackle this issue, we propose a learning algorithm that decouples the action constraints from the policy parameter update by leveraging state-wise Frank-Wolfe and a regression-based policy update scheme. Moreover, we show that the proposed algorithm enjoys convergence and policy improvement properties in the tabular case as well as generalizes the popular DDPG algorithm for action-constrained RL in the general case. Through experiments, we demonstrate that the proposed algorithm significantly outperforms the benchmark methods on a variety of control tasks.
arXiv commentPublished in UAI 2021
References in corpus (15)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Safe Exploration in Continuous Action Spaces
- Reward Constrained Policy Optimization
- Differentiable Convex Optimization Layers
- On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
- Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
- Convergence Rate of Frank-Wolfe for Non-Convex Objectives
- Projection-Based Constrained Policy Optimization
- ChainerRL: A Deep Reinforcement Learning Library
- Clipped Action Policy Gradient
- Internet Congestion Control via Deep Reinforcement Learning
- Constrained Markov Decision Processes via Backward Value Functions
- Resource Constrained Deep Reinforcement Learning