Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures

Abstract

Model-free or learning-based control, in particular, reinforcement learning (RL), is expected to be applied for complex robotic tasks. Traditional RL requires a policy to be optimized is state-dependent, that means, the policy is a kind of feedback (FB) controllers. Due to the necessity of correct state observation in such a FB controller, it is sensitive to sensing failures. To alleviate this drawback of the FB controllers, feedback error learning integrates one of them with a feedforward (FF) controller. RL can be improved by dealing with the FB/FF policies, but to the best of our knowledge, a methodology for learning them in a unified manner has not been developed. In this paper, we propose a new optimization problem for optimizing both the FB/FF policies simultaneously. Inspired by control as inference, the optimization problem considers minimization/maximization of divergences between trajectory, predicted by the composed policy and a stochastic dynamics model, and optimal/non-optimal trajectories. By approximating the stochastic dynamics model using variational method, we naturally derive a regularization between the FB/FF policies. In numerical simulations and a robot experiment, we verified that the proposed method can stably optimize the composed policy even with the different learning law from the traditional RL. In addition, we demonstrated that the FF policy is robust to the sensing failures and can hold the optimal motion. Attached video is also uploaded on youtube: https://youtu.be/zLL4uXIRmrE

23 pages, 12 figures

References in corpus (9)

• Recurrent Neural Network Regularization
• t-Soft Update of Target Network for Deep Reinforcement Learning
• Model-Augmented Actor-Critic: Backpropagating through Paths
• How to Train Your Differentiable Filter
• Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization
• Latent Representation in Human-Robot Interaction with Explicit Consideration of Periodic Dynamics
• Proximal Policy Optimization with Relative Pearson Divergence
• Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learning
• Towards Deep Robot Learning with Optimizer applicable to Non-stationary Problems

Cited by in corpus (2)

• Latent Representation in Human-Robot Interaction with Explicit Consideration of Periodic Dynamics
• Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency