Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy
arXiv:2012.14098
Abstract
While deep reinforcement learning has achieved tremendous successes in various applications, most existing works only focus on maximizing the expected value of total return and thus ignore its inherent stochasticity. Such stochasticity is also known as the aleatoric uncertainty and is closely related to the notion of risk. In this work, we make the first attempt to study risk-sensitive deep reinforcement learning under the average reward setting with the variance risk criteria. In particular, we focus on a variance-constrained policy optimization problem where the goal is to find a policy that maximizes the expected value of the long-run average reward, subject to a constraint that the long-run variance of the average reward is upper bounded by a threshold. Utilizing Lagrangian and Fenchel dualities, we transform the original problem into an unconstrained saddle-point policy optimization problem, and propose an actor-critic algorithm that iteratively and efficiently updates the policy, the Lagrange multiplier, and the Fenchel dual variable. When both the value and policy functions are represented by multi-layer overparameterized neural networks, we prove that our actor-critic algorithm generates a sequence of policies that finds a globally optimal policy at a sublinear rate. Further, We provide numerical studies of the proposed method using two real datasets to back up the theoretical results.
arXiv comment59 pages
References in corpus (17)
- Trust Region Policy Optimization
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Reinforcement Learning with Deep Energy-Based Policies
- A Theoretical Analysis of Deep Q-Learning
- Language Understanding for Text-based Games Using Deep Reinforcement Learning
- Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
- On the Global Convergence Rates of Softmax Policy Gradient Methods
- Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy
- Convergence of Adversarial Training in Overparametrized Neural Networks
- Estimating Risk and Uncertainty in Deep Reinforcement Learning
- Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization
- On the Convergence Rate of Stochastic Mirror Descent for Nonsmooth Nonconvex Optimization
- Variance Adjusted Actor Critic Algorithms
- A Block Coordinate Ascent Algorithm for Mean-Variance Optimization
- Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
- Risk-Averse Learning by Temporal Difference Methods