Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
arXiv:2012.04839
Abstract
In reinforcement learning, domain randomisation is an increasingly popular technique for learning more general policies that are robust to domain-shifts at deployment. However, naively aggregating information from randomised domains may lead to high variance in gradient estimation and unstable learning process. To address this issue, we present a peer-to-peer online distillation strategy for RL termed P2PDRL, where multiple workers are each assigned to a different environment, and exchange knowledge through mutual regularisation based on Kullback-Leibler divergence. Our experiments on continuous control tasks show that P2PDRL enables robust learning across a wider randomisation distribution than baselines, and more robust generalisation to new environments at testing.
References in corpus (10)
- Distilling the Knowledge in a Neural Network
- Emergence of Locomotion Behaviours in Rich Environments
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- Assessing Generalization in Deep Reinforcement Learning
- Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
- Sim-to-Real Robot Learning from Pixels with Progressive Nets
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Policy Distillation
- Investigating Generalisation in Continuous Deep Reinforcement Learning
- Been There, Done That: Meta-Learning with Episodic Recall