Preparing for the Unknown: Learning a Universal Policy with Online System Identification
arXiv:1702.02453
Abstract
We present a new method of learning control policies that successfully operate under unknown dynamic models. We create such policies by leveraging a large number of training examples that are generated using a physical simulator. Our system is made of two components: a Universal Policy (UP) and a function for Online System Identification (OSI). We describe our control policy as universal because it is trained over a wide array of dynamic models. These variations in the dynamic model may include differences in mass and inertia of the robots' components, variable friction coefficients, or unknown mass of an object to be manipulated. By training the Universal Policy with this variation, the control policy is prepared for a wider array of possible conditions when executed in an unknown environment. The second part of our system uses the recent state and action history of the system to predict the dynamics model parameters mu. The value of mu from the Online System Identification is then provided as input to the control policy (along with the system state). Together, UP-OSI is a robust control policy that can be used across a wide range of dynamic models, and that is also responsive to sudden changes in the environment. We have evaluated the performance of this system on a variety of tasks, including the problem of cart-pole swing-up, the double inverted pendulum, locomotion of a hopper, and block-throwing of a manipulator. UP-OSI is effective at these tasks across a wide range of dynamic models. Moreover, when tested with dynamic models outside of the training range, UP-OSI outperforms the Universal Policy alone, even when UP is given the actual value of the model dynamics. In addition to the benefits of creating more robust controllers, UP-OSI also holds out promise of narrowing the Reality Gap between simulated and real physical systems.
Accepted as a conference paper at RSS 2017
References in corpus (4)
Cited by in corpus (24)
- Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
- A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
- Transfer Learning in Deep Reinforcement Learning: A Survey
- Adaptive Guidance and Integrated Navigation with Reinforcement Meta-Learning
- Assessing Transferability from Simulation to Reality for Reinforcement Learning
- Sim2Real View Invariant Visual Servoing by Recurrent Control
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
- DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions
- Adaptive Guidance with Reinforcement Meta-Learning
- Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
- GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model
- Map-based Multi-Policy Reinforcement Learning: Enhancing Adaptability of Robots by Deep Reinforcement Learning
- One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
- Evaluating the Robustness of Collaborative Agents
- ADAIL: Adaptive Adversarial Imitation Learning
- Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
- Auto-Tuned Sim-to-Real Transfer
- Unsupervised Visual Attention and Invariance for Reinforcement Learning
- MANGA: Method Agnostic Neural-policy Generalization and Adaptation
- Multi-task Learning with Gradient Guided Policy Specialization
- Zero-shot Deep Reinforcement Learning Driving Policy Transfer for Autonomous Vehicles based on Robust Control
- Not Only Domain Randomization: Universal Policy with Embedding System Identification
- DROID: Minimizing the Reality Gap using Single-Shot Human Demonstration
- Fast Motion Planning for High-DOF Robot Systems Using Hierarchical System Identification