Deep Reinforcement Learning with Robust and Smooth Policy

Abstract

Deep reinforcement learning (RL) has achieved great empirical successes in various domains. However, the large search space of neural networks requires a large amount of data, which makes the current RL algorithms not sample efficient. Motivated by the fact that many environments with continuous state space have smooth transitions, we propose to learn a smooth policy that behaves smoothly with respect to states. We develop a new framework -- \textbf{S}mooth \textbf{R}egularized \textbf{R}einforcement \textbf{L}earning (${\text{SR}}^2\text{L}$), where the policy is trained with smoothness-inducing regularization. Such regularization effectively constrains the search space, and enforces smoothness in the learned policy. Moreover, our proposed framework can also improve the robustness of policy against measurement error in the state space, and can be naturally extended to distribubutionally robust setting. We apply the proposed framework to both on-policy (TRPO) and off-policy algorithm (DDPG). Through extensive experiments, we demonstrate that our method achieves improved sample efficiency and robustness.

arXiv commentICML 2020

References in corpus (9)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Unsupervised Data Augmentation for Consistency Training
• Understanding deep learning requires rethinking generalization
• Theoretically Principled Trade-off between Robustness and Accuracy
• Robust Adversarial Reinforcement Learning
• Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks