Value constrained model-free continuous control

Abstract

The naive application of Reinforcement Learning algorithms to continuous control problems -- such as locomotion and manipulation -- often results in policies which rely on high-amplitude, high-frequency control signals, known colloquially as bang-bang control. Although such solutions may indeed maximize task reward, they can be unsuitable for real world systems. Bang-bang control may lead to increased wear and tear or energy consumption, and tends to excite undesired second-order dynamics. To counteract this issue, multi-objective optimization can be used to simultaneously optimize both the reward and some auxiliary cost that discourages undesired (e.g. high-amplitude) control. In principle, such an approach can yield the sought after, smooth, control policies. It can, however, be hard to find the correct trade-off between cost and return that results in the desired behavior. In this paper we propose a new constraint-based reinforcement learning approach that ensures task success while minimizing one or more auxiliary costs (such as control effort). We employ Lagrangian relaxation to learn both (a) the parameters of a control policy that satisfies the desired constraints and (b) the Lagrangian multipliers for the optimization. Moreover, we demonstrate that we can satisfy constraints either in expectation or in a per-step fashion, and can even learn a single policy that is able to dynamically trade-off between return and cost. We demonstrate the efficacy of our approach using a number of continuous control benchmark tasks, a realistic, energy-optimized quadruped locomotion task, as well as a reaching task on a real robot arm.

References in corpus (4)

• Trust Region Policy Optimization
• DeepMind Control Suite
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Multi-Objective Deep Reinforcement Learning

Cited by in corpus (13)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Challenges of Real-World Reinforcement Learning
• ACES -- Automatic Configuration of Energy Harvesting Sensors with Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• Safe Reinforcement Learning Using Black-Box Reachability Analysis
• Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach
• On the model-based stochastic value gradient for continuous reinforcement learning
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• Augmenting learning using symmetry in a biologically-inspired domain
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies
• On the Volatility of Optimal Control Policies and the Capacity of a Class of Linear Quadratic Regulators