Adversarially Regularized Policy Learning Guided by Trajectory Optimization
arXiv:2109.07627
Abstract
Recent advancement in combining trajectory optimization with function approximation (especially neural networks) shows promise in learning complex control policies for diverse tasks in robot systems. Despite their great flexibility, the large neural networks for parameterizing control policies impose significant challenges. The learned neural control policies are often overcomplex and non-smooth, which can easily cause unexpected or diverging robot motions. Therefore, they often yield poor generalization performance in practice. To address this issue, we propose adVErsarially Regularized pOlicy learNIng guided by trajeCtory optimizAtion (VERONICA) for learning smooth control policies. Specifically, our proposed approach controls the smoothness (local Lipschitz continuity) of the neural control policies by stabilizing the output control with respect to the worst-case perturbation to the input state. Our experiments on robot manipulation show that our proposed approach not only improves the sample efficiency of neural policy learning but also enhances the robustness of the policy against various types of disturbances, including sensor noise, environmental uncertainty, and model mismatch.
arXiv commentAccepted at L4DC 2022
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Decoupled Weight Decay Regularization
- Unsupervised Data Augmentation for Consistency Training
- Theoretically Principled Trade-off between Robustness and Accuracy
- Generalized Inner Loop Meta-Learning
- A Closer Look at Accuracy vs. Robustness
- Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
- Task-Relevant Adversarial Imitation Learning
- Online Trajectory Planning Through Combined Trajectory Optimization and Function Approximation: Application to the Exoskeleton Atalante
- On Value Discrepancy of Imitation Learning
- SyDeBO: Symbolic-Decision-Embedded Bilevel Optimization for Long-Horizon Manipulation in Dynamic Environments