Understanding Domain Randomization for Sim-to-real Transfer
arXiv:2110.03239
Abstract
Reinforcement learning encounters many challenges when applied directly in the real world. Sim-to-real transfer is widely used to transfer the knowledge learned from simulation to the real world. Domain randomization -- one of the most popular algorithms for sim-to-real transfer -- has been demonstrated to be effective in various tasks in robotics and autonomous driving. Despite its empirical successes, theoretical understanding on why this simple algorithm works is limited. In this paper, we propose a theoretical framework for sim-to-real transfers, in which the simulator is modeled as a set of MDPs with tunable parameters (corresponding to unknown physical parameters such as friction). We provide sharp bounds on the sim-to-real gap -- the difference between the value of policy returned by domain randomization and the value of an optimal policy for the real world. We prove that sim-to-real transfer can succeed under mild conditions without any real-world training samples. Our theory also highlights the importance of using memory (i.e., history-dependent policies) in domain randomization. Our proof is based on novel techniques that reduce the problem of bounding the sim-to-real gap to the problem of designing efficient learning algorithms for infinite-horizon MDPs, which we believe are of independent interest.
References in corpus (17)
- Playing Atari with Deep Reinforcement Learning
- Provably Efficient Reinforcement Learning with Linear Function Approximation
- Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
- REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs
- Sim2real transfer learning for 3D human pose estimation: motion to the rescue
- Policy Transfer with Strategy Optimization
- Model-based Reinforcement Learning and the Eluder Dimension
- Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension
- Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms
- Provably Efficient Q-Learning with Low Switching Cost
- Modeling Long-horizon Tasks as Sequential Interaction Landscapes
- Improved Analysis of UCRL2 with Empirical Bernstein Inequality
- Sample-Efficient Reinforcement Learning of Undercomplete POMDPs
- RL for Latent MDPs: Regret Guarantees and a Lower Bound
- How Does an Approximate Model Help in Reinforcement Learning?
- PAC Reinforcement Learning without Real-World Feedback
- Online Sub-Sampling for Reinforcement Learning with General Function Approximation