FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control

Abstract

In recent years significant progress has been made in dealing with challenging problems using reinforcement learning.Despite its great success, reinforcement learning still faces challenge in continuous control tasks. Conventional methods always compute the derivatives of the optimal goal with a costly computation resources, and are inefficient, unstable and lack of robust-ness when dealing with such tasks. Alternatively, derivative-based methods treat the optimization process as a blackbox and show robustness and stability in learning continuous control tasks, but not data efficient in learning. The combination of both methods so as to get the best of the both has raised attention. However, most of the existing combination works adopt complex neural networks (NNs) as the policy for control. The double-edged sword of deep NNs can yield better performance, but also makes it difficult for parameter tuning and computation. To this end, in this paper we presents a novel method called FiDi-RL, which incorporates deep RL with Finite-Difference (FiDi) policy search.FiDi-RL combines Deep Deterministic Policy Gradients (DDPG)with Augment Random Search (ARS) and aims at improving the data efficiency of ARS. The empirical results show that FiDi-RL can improves the performance and stability of ARS, and provide competitive results against some existing deep reinforcement learning methods

arXiv commentI found some theoretical errors

References in corpus (17)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Counterfactual Multi-Agent Policy Gradients
• Parameter Space Noise for Exploration
• PEGASUS: A Policy Search Method for Large MDPs and POMDPs
• Deep Reinforcement Learning that Matters
• A Deeper Look at Experience Replay
• Path Integral Policy Improvement with Covariance Matrix Adaptation
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Evolution-Guided Policy Gradient in Reinforcement Learning
• Guided evolutionary strategies: Augmenting random search with surrogate gradients