Reinforcement Learning with Convex Constraints

Abstract

In standard reinforcement learning (RL), a learning agent seeks to optimize the overall reward. However, many key aspects of a desired behavior are more naturally expressed as constraints. For instance, the designer may want to limit the use of unsafe actions, increase the diversity of trajectories to enable exploration, or approximate expert trajectories when rewards are sparse. In this paper, we propose an algorithmic scheme that can handle a wide class of constraints in RL tasks: specifically, any constraints that require expected values of some vector measurements (such as the use of an action) to lie in a convex set. This captures previously studied constraints (such as safety and proximity to an expert), but also enables new classes of constraints (such as diversity). Our approach comes with rigorous theoretical guarantees and only relies on the ability to approximately solve standard RL tasks. As a result, it can be easily adapted to work with any model-free or model-based RL. In our experiments, we show that it matches previous algorithms that enforce safety via constraints, but can also enforce new properties that these algorithms do not incorporate, such as diversity.

References in corpus (2)

• Reward Constrained Policy Optimization
• Provably Efficient Maximum Entropy Exploration

Cited by in corpus (15)

• Reward-Free Exploration for Reinforcement Learning
• On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method
• Lexicographic Multi-Objective Reinforcement Learning
• Constrained episodic reinforcement learning in concave-convex and knapsack settings
• Nearly Minimax Optimal Reward-free Reinforcement Learning
• Reinforcement Learning Your Way: Agent Characterization through Policy Regularization
• Deep Reinforcement Learning for Demand Driven Services in Logistics and Transportation Systems: A Survey
• Inverse Constrained Reinforcement Learning
• Interactive Learning from Activity Description
• Reward is enough for convex MDPs
• Model-Free Algorithm and Regret Analysis for MDPs with Long-Term Constraints
• Time-Dependent Blackwell Approachability and Application to Absorbing Games
• Reinforcement Learning with Intrinsic Affinity for Personalized Prosperity Management
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• A Policy Efficient Reduction Approach to Convex Constrained Deep Reinforcement Learning