Constrained Upper Confidence Reinforcement Learning

Abstract

Constrained Markov Decision Processes are a class of stochastic decision problems in which the decision maker must select a policy that satisfies auxiliary cost constraints. This paper extends upper confidence reinforcement learning for settings in which the reward function and the constraints, described by cost functions, are unknown a priori but the transition kernel is known. Such a setting is well-motivated by a number of applications including exploration of unknown, potentially unsafe, environments. We present an algorithm C-UCRL and show that it achieves sub-linear regret ($O(T^{\frac{3}{4}}\sqrt{\log (T/\delta )})$) with respect to the reward while satisfying the constraints even while learning with probability $1-\delta$. Illustrative examples are provided.

References in corpus (6)

• Continuous control with deep reinforcement learning
• Safe Model-based Reinforcement Learning with Stability Guarantees
• AI Safety Gridworlds
• Constrained Policy Optimization
• A Lyapunov-based Approach to Safe Reinforcement Learning
• Budget-Constrained Multi-Armed Bandits with Multiple Plays

Cited by in corpus (9)

• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization
• Exploration-Exploitation in Constrained MDPs
• Constrained episodic reinforcement learning in concave-convex and knapsack settings
• Upper Confidence Primal-Dual Reinforcement Learning for CMDP with Adversarial Loss
• Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret
• Provably Efficient Model-Free Algorithm for MDPs with Peak Constraints
• Efficient Reinforcement Learning in Factored MDPs with Application to Constrained RL
• Safe Reinforcement Learning with Linear Function Approximation
• Online Selection of Diverse Committees