Benchmarking Potential Based Rewards for Learning Humanoid Locomotion

Abstract

The main challenge in developing effective reinforcement learning (RL) pipelines is often the design and tuning the reward functions. Well-designed shaping reward can lead to significantly faster learning. Naively formulated rewards, however, can conflict with the desired behavior and result in overfitting or even erratic performance if not properly tuned. In theory, the broad class of potential based reward shaping (PBRS) can help guide the learning process without affecting the optimal policy. Although several studies have explored the use of potential based reward shaping to accelerate learning convergence, most have been limited to grid-worlds and low-dimensional systems, and RL in robotics has predominantly relied on standard forms of reward shaping. In this paper, we benchmark standard forms of shaping with PBRS for a humanoid robot. We find that in this high-dimensional system, PBRS has only marginal benefits in convergence speed. However, the PBRS reward terms are significantly more robust to scaling than typical reward shaping approaches, and thus easier to tune.

References in corpus (11)

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Learning robust perceptive locomotion for quadrupedal robots in the wild
• Automatic LQR Tuning Based on Gaussian Process Global Optimization
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Potential-Based Shaping and Q-Value Initialization are Equivalent
• Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning
• Learning Agile Robotic Locomotion Skills by Imitating Animals
• Learning to Brachiate via Simplified Model Imitation
• Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning
• Learning to Run with Potential-Based Reward Shaping and Demonstrations from Video Data

Cited by in corpus (4)

• HumanMimic: Learning Natural Locomotion and Transitions for Humanoid Robot via Wasserstein Adversarial Imitation
• Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulator
• Efficient Reinforcement Learning for Jumping Monopods
• Humanoid Robot Running Through Random Stepping Stones and Jumping Over Obstacles: Step Adaptation Using Spring-Mass Trajectories