S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning

Abstract

Offline reinforcement learning proposes to learn policies from large collected datasets without interacting with the physical environment. These algorithms have made it possible to learn useful skills from data that can then be deployed in the environment in real-world settings where interactions may be costly or dangerous, such as autonomous driving or factories. However, current algorithms overfit to the dataset they are trained on and exhibit poor out-of-distribution generalization to the environment when deployed. In this paper, we study the effectiveness of performing data augmentations on the state space, and study 7 different augmentation schemes and how they behave with existing offline RL algorithms. We then combine the best data performing augmentation scheme with a state-of-the-art Q-learning technique, and improve the function approximation of the Q-networks by smoothening out the learned state-action space. We experimentally show that using this Surprisingly Simple Self-Supervision technique in RL (S4RL), we significantly improve over the current state-of-the-art algorithms on offline robot learning environments such as MetaWorld [1] and RoboSuite [2,3], and benchmark datasets such as D4RL [4].

References in corpus (23)

• mixup: Beyond Empirical Risk Minimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Conservative Q-Learning for Offline Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Reinforcement Learning with Augmented Data
• robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Experience Replay with Likelihood-free Importance Weights
• RoboNet: Large-Scale Multi-Robot Learning
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• D2RL: Deep Dense Architectures in Reinforcement Learning

Cited by in corpus (4)

• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Implicit Behavioral Cloning
• Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics