SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations

Abstract

Robotic agents trained using reinforcement learning have the problem of taking unreliable actions in an out-of-distribution (OOD) state. Agents can easily become OOD in real-world environments because it is almost impossible for them to visit and learn the entire state space during training. Unfortunately, unreliable actions do not ensure that agents perform their original tasks successfully. Therefore, agents should be able to recognize whether they are in OOD states and learn how to return to the learned state distribution rather than continue to take unreliable actions. In this study, we propose a novel method for retraining agents to recover from OOD situations in a self-supervised manner when they fall into OOD states. Our in-depth experimental results demonstrate that our method substantially improves the agent's ability to recover from OOD situations in terms of sample efficiency and restoration of the performance for the original tasks. Moreover, we show that our method can retrain the agent to recover from OOD situations even when in-distribution states are difficult to visit through exploration.

9 pages, 5 figures. Proceedings of the 32nd International Joint Conference on Artificial Intelligence, 2023

References in corpus (7)

• Emergence of Locomotion Behaviours in Rich Environments
• Solving Rubik's Cube with a Robot Hand
• Conservative Q-Learning for Offline Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning