Reinforcement Learning by Guided Safe Exploration
arXiv:2307.14316 · doi:10.3233/FAIA230598
Abstract
Safety is critical to broadening the application of reinforcement learning (RL). Often, we train RL agents in a controlled environment, such as a laboratory, before deploying them in the real world. However, the real-world target task might be unknown prior to deployment. Reward-free RL trains an agent without the reward to adapt quickly once the reward is revealed. We consider the constrained reward-free setting, where an agent (the guide) learns to explore safely without the reward signal. This agent is trained in a controlled environment, which allows unsafe interactions and still provides the safety signal. After the target task is revealed, safety violations are not allowed anymore. Thus, the guide is leveraged to compose a safe behaviour policy. Drawing from transfer learning, we also regularize a target policy (the student) towards the guide while the student is unreliable and gradually eliminate the influence of the guide as training progresses. The empirical analysis shows that this method can achieve safe transfer learning and helps the student solve the target task faster.
arXiv commentAccecpted at ECAI 2023
References in corpus (30)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Soft Actor-Critic Algorithms and Applications
- Reinforcement Learning with Deep Energy-Based Policies
- Reward Constrained Policy Optimization
- Transfer Learning in Deep Reinforcement Learning: A Survey
- Near Optimal Behavior via Approximate State Abstraction
- Provably Efficient Maximum Entropy Exploration
- Diversity is All You Need: Learning Skills without a Reward Function
- Learning to Walk in the Real World with Minimal Human Effort
- Efficient Exploration via State Marginal Matching
- Projection-Based Constrained Policy Optimization
- Risk-Constrained Reinforcement Learning with Percentile Risk Criteria
- Safe Reinforcement Learning via Curriculum Induction
- Maximum Entropy RL (Provably) Solves Some Robust RL Problems
- Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
- State Entropy Maximization with Random Encoders for Efficient Exploration
- Learning latent state representation for speeding up exploration
- Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
- Curriculum Learning for Safe Mapless Navigation
- Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
- Risk-Aware Transfer in Reinforcement Learning using Successor Features
- One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
- Marginalized State Distribution Entropy Regularization in Policy Optimization
- A Simple Reward-free Approach to Constrained Reinforcement Learning
- MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance
- Safe Driving via Expert Guided Policy Optimization
- Discovering Diverse Nearly Optimal Policies with Successor Features
- Safe Reinforcement Learning via Shielding under Partial Observability
- Density Constrained Reinforcement Learning