Safe Reinforcement Learning with Natural Language Constraints
arXiv:2010.05150
Abstract
While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propose learning to interpret natural language constraints for safe RL. To this end, we first introduce HazardWorld, a new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We then develop an agent with a modular architecture that can interpret and adhere to such textual constraints while learning new tasks. Our model consists of (1) a constraint interpreter that encodes textual constraints into spatial and temporal representations of forbidden states, and (2) a policy network that uses these representations to produce a policy achieving minimal constraint violations during training. Across different domains in HazardWorld, we show that our method achieves higher rewards (up to11x) and fewer constraint violations (by 1.8x) compared to existing approaches. However, in terms of absolute performance, HazardWorld still poses significant challenges for agents to learn efficiently, motivating the need for future work.
arXiv commentThe first two authors contributed equally
References in corpus (17)
- Trust Region Policy Optimization
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Benchmarking Deep Reinforcement Learning for Continuous Control
- Safe Model-based Reinforcement Learning with Stability Guarantees
- Reward Constrained Policy Optimization
- Challenges of Real-World Reinforcement Learning
- Speaker-Follower Models for Vision-and-Language Navigation
- Benchmarking Batch Deep Reinforcement Learning Algorithms
- Lyapunov-based Safe Policy Optimization for Continuous Control
- Reinforcement Learning from Imperfect Demonstrations
- Projection-Based Constrained Policy Optimization
- BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
- From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
- Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction
- Safe Reinforcement Learning via Curriculum Induction
- Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
- Learning a natural-language to LTL executable semantic parser for grounded robotics