LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World

Abstract

Model-based reinforcement learning has attracted much attention due to its high sample efficiency and is expected to be applied to real-world robotic applications. In the real world, as unobservable disturbances can lead to unexpected situations, robot policies should be taken to improve not only control performance but also robustness. Adversarial learning is an effective way to improve robustness, but excessive adversary would increase the risk of malfunction, and make the control performance too conservative. Therefore, this study addresses a new adversarial learning framework to make reinforcement learning robust moderately and not conservative too much. To this end, the adversarial learning is first rederived with variational inference. In addition, \textit{light robustness}, which allows for maximizing robustness within an acceptable performance degradation, is utilized as a constraint. As a result, the proposed framework, so-called LiRA, can automatically adjust adversary level, balancing robustness and conservativeness. The expected behaviors of LiRA are confirmed in numerical simulations. In addition, LiRA succeeds in learning a force-reactive gait control of a quadrupedal robot only with real-world data collected less than two hours.

21 pages, 17 figures (accepted in Robotics and Autonomous Systems)

References in corpus (11)

• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• Safe Reinforcement Learning Using Robust MPC
• Root Mean Square Layer Normalization
• AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization
• AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows
• Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs
• Squareplus: A Softplus-Like Algebraic Rectifier
• Sparse Representation Learning with Modified q-VAE towards Minimal Realization of World Model
• Real-time Sampling-based Model Predictive Control based on Reverse Kullback-Leibler Divergence and Its Adaptive Acceleration
• Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
• Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint