MPC-Inspired Neural Network Policies for Sequential Decision Making

Abstract

In this paper we investigate the use of MPC-inspired neural network policies for sequential decision making. We introduce an extension to the DAgger algorithm for training such policies and show how they have improved training performance and generalization capabilities. We take advantage of this extension to show scalable and efficient training of complex planning policy architectures in continuous state and action spaces. We provide an extensive comparison of neural network policies by considering feed forward policies, recurrent policies, and recurrent policies with planning structure inspired by the Path Integral control framework. Our results suggest that MPC-type recurrent policies have better robustness to disturbances and modeling error.

arXiv commentFixed missing reference to section 4.1

References in corpus (4)

• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning
• Prediction and Control with Temporal Segment Models

Cited by in corpus (13)

• Differentiable MPC for End-to-end Planning and Control
• Universal Planning Networks
• Model-Augmented Actor-Critic: Backpropagating through Paths
• Safe end-to-end imitation learning for model predictive control
• On the model-based stochastic value gradient for continuous reinforcement learning
• Learning to Optimize in Model Predictive Control
• Neural Lyapunov Differentiable Predictive Control
• Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
• NOVAS: Non-convex Optimization via Adaptive Stochastic Search for End-to-End Learning and Control
• Cloth Manipulation Planning on Basis of Mesh Representations with Incomplete Domain Knowledge and Voxel-to-Mesh Estimation
• Runtime-Safety-Guided Policy Repair
• AReN: Assured ReLU NN Architecture for Model Predictive Control of LTI Systems
• Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning