Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challenge

Abstract

This paper presents a deep learning framework that is capable of solving partially observable locomotion tasks based on our novel interpretation of Recurrent Deterministic Policy Gradient (RDPG). We study on bias of sampled error measure and its variance induced by the partial observability of environment and subtrajectory sampling, respectively. Three major improvements are introduced in our RDPG based learning framework: tail-step bootstrap of interpolated temporal difference, initialisation of hidden state using past trajectory scanning, and injection of external experiences learned by other agents. The proposed learning framework was implemented to solve the Bipedal-Walker challenge in OpenAI's gym simulation environment where only partial state information is available. Our simulation study shows that the autonomous behaviors generated by the RDPG agent are highly adaptive to a variety of obstacles and enables the agent to effectively traverse rugged terrains for long distance with higher success rate than leading contenders.

arXiv commentPublished in IEEE proceedings: 2018 15th International Conference on Control, Automation, Robotics and Vision (IEEE-ICARCV)

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Emergence of Locomotion Behaviours in Rich Environments
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Deep Reinforcement Learning: An Overview
• Parameter Space Noise for Exploration
• Memory-based control with recurrent neural networks
• A Deeper Look at Experience Replay
• Playing FPS Games with Deep Reinforcement Learning
• Training recurrent networks online without backtracking
• Emergence of Human-comparable Balancing Behaviors by Deep Reinforcement Learning
• Investigating Recurrence and Eligibility Traces in Deep Q-Networks

Cited by in corpus (7)

• Bipedal Walking Robot using Deep Deterministic Policy Gradient
• Dynamic Control of a Fiber Manufacturing Process using Deep Reinforcement Learning
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• Policy Optimization Reinforcement Learning with Entropy Regularization
• First-Order Preconditioning via Hypergradient Descent
• Reinforcement learning with world model
• Learning Vision-Guided Dynamic Locomotion Over Challenging Terrains