Gaussian Processes for Data-Efficient Learning in Robotics and Control

Abstract

Autonomous learning has been a promising direction in control and robotics for more than a decade since data-driven learning allows to reduce the amount of engineering knowledge, which is otherwise required. However, autonomous reinforcement learning (RL) approaches typically require many interactions with the system to learn controllers, which is a practical limitation in real systems, such as robots, where many interactions can be impractical and time consuming. To address this problem, current learning approaches typically require task-specific knowledge in form of expert demonstrations, realistic simulators, pre-shaped policies, or specific knowledge about the underlying dynamics. In this article, we follow a different approach and speed up learning by extracting more information from data. In particular, we learn a probabilistic, non-parametric Gaussian process transition model of the system. By explicitly incorporating model uncertainty into long-term planning and controller learning our approach reduces the effects of model errors, a key problem in model-based learning. Compared to state-of-the art RL our model-based policy search method achieves an unprecedented speed of learning. We demonstrate its applicability to autonomous learning in real robot and control tasks.

20 pages, 29 figures; fixed a typo in equation on page 8

References in corpus (2)

• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• PEGASUS: A Policy Search Method for Large MDPs and POMDPs

Cited by in corpus (112)

• An Algorithmic Perspective on Imitation Learning
• Benchmarking Model-Based Reinforcement Learning
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Reset-free Trial-and-Error Learning for Robot Damage Recovery
• Data-efficient Auto-tuning with Bayesian Optimization: An Industrial Control Study
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning
• Variational Fourier features for Gaussian processes
• Autonomous robotic nanofabrication with reinforcement learning
• Active Learning in Robotics: A Review of Control Principles
• Residual Policy Learning
• Nested Kriging predictions for datasets with large number of observations
• Linear Maximum Margin Classifier for Learning from Uncertain Data
• Towards a Common Implementation of Reinforcement Learning for Multiple Robotic Tasks
• Learning to Guide: Guidance Law Based on Deep Meta-learning and Model Predictive Path Integral Control
• Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics
• Guaranteed Coverage Prediction Intervals with Gaussian Process Regression
• Exact Gaussian Processes on a Million Data Points
• Goal-Driven Dynamics Learning via Bayesian Optimization
• On Policy Learning Robust to Irreversible Events: An Application to Robotic In-Hand Manipulation
• Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning
• Model-Based Policy Search Using Monte Carlo Gradient Estimation with Real Systems Application
• Personalized Optimization with User's Feedback
• Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming
• Matérn Gaussian processes on Riemannian manifolds
• Data-driven Aerodynamic Analysis of Structures using Gaussian Processes
• Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models
• Stable Gaussian Process based Tracking Control of Lagrangian Systems
• MBMF: Model-Based Priors for Model-Free Reinforcement Learning
• Semi-described and semi-supervised learning with Gaussian processes
• Exploration of the Applicability of Probabilistic Inference for Learning Control in Underactuated Autonomous Underwater Vehicles
• Controlling Robot Morphology from Incomplete Measurements
• RoboNet: Large-Scale Multi-Robot Learning
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• Constant-Time Predictive Distributions for Gaussian Processes
• Sample Efficient Path Integral Control under Uncertainty
• Data-efficient Model Learning and Prediction for Contact-rich Manipulation Tasks
• Partitioned Active Learning for Heterogeneous Systems
• Model-based Reinforcement Learning from Signal Temporal Logic Specifications
• Prediction performance after learning in Gaussian process regression
• Federated Gaussian Process: Convergence, Automatic Personalization and Multi-fidelity Modeling
• Variable impedance control and learning -- A review
• GaPT: Gaussian Process Toolkit for Online Regression with Application to Learning Quadrotor Dynamics
• Efficiently Sampling Functions from Gaussian Process Posteriors
• Physically Consistent Learning of Conservative Lagrangian Systems with Gaussian Processes
• Convergence Guarantees for Gaussian Process Means With Misspecified Likelihoods and Smoothness
• Convergence results for an averaged LQR problem with applications to reinforcement learning
• Identification of Gaussian Process State Space Models
• Safe Interactive Model-Based Learning
• A Survey of Behavior Learning Applications in Robotics -- State of the Art and Perspectives
• Composite likelihood estimation for a gaussian process under fixed domain asymptotics
• A Bayesian Approach to Policy Recognition and State Representation Learning
• Micro-Data Learning: The Other End of the Spectrum
• Tuneful: An Online Significance-Aware Configuration Tuner for Big Data Analytics
• Stable Model-based Control with Gaussian Process Regression for Robot Manipulators
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• On the Universal Transformation of Data-Driven Models to Control Systems
• Reinforcement Learning Ship Autopilot: Sample efficient and Model Predictive Control-based Approach
• Computationally Efficient Bayesian Learning of Gaussian Process State Space Models
• Adaptive Probabilistic Trajectory Optimization via Efficient Approximate Inference
• Safe Learning of Quadrotor Dynamics Using Barrier Certificates
• Data-driven Policy Transfer with Imprecise Perception Simulation
• Incremental Nonlinear Stability Analysis of Stochastic Systems Perturbed by Lévy Noise
• Model Imitation for Model-Based Reinforcement Learning
• Conditional Neural Expert Processes for Learning Movement Primitives from Demonstration
• Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning
• Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
• Learning ODE Models with Qualitative Structure Using Gaussian Processes
• Model-based Path Integral Stochastic Control: A Bayesian Nonparametric Approach
• Pathwise Conditioning of Gaussian Processes
• Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
• Inference for Gaussian Processes with Matérn Covariogram on Compact Riemannian Manifolds
• Transfer Learning Across Patient Variations with Hidden Parameter Markov Decision Processes
• Aggregating Dependent Gaussian Experts in Local Approximation
• Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systems
• Self-learning and adaptation in a sensorimotor framework
• Localized active learning of Gaussian process state space models
• Meta Learning MPC using Finite-Dimensional Gaussian Process Approximations
• Multi-group Gaussian Processes
• Forethought and Hindsight in Credit Assignment
• Learning Unmanned Aerial Vehicle Control for Autonomous Target Following
• Learning Contact Dynamics using Physically Structured Neural Networks
• Safe Policy Improvement in Constrained Markov Decision Processes
• Reference design for closed loop system optimization
• Gaussian Process Uniform Error Bounds with Unknown Hyperparameters for Safety-Critical Applications
• Black-Box Data-efficient Policy Search for Robotics
• OTTR: Off-Road Trajectory Tracking using Reinforcement Learning
• Planning under Uncertainty to Goal Distributions
• Gaussian Experts Selection using Graphical Models
• Learning-based Event-triggered MPC with Gaussian processes under terminal constraints
• Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes
• Deep Sigma Point Processes for RCS Modeling in Spaceborne SAR Imagery
• Deep Gaussian Covariance Network with Trajectory Sampling for Data-Efficient Policy Search
• Learning Nonparametric Volterra Kernels with Gaussian Processes
• Learning self-triggered controllers with Gaussian processes
• Reinforcement Learning for Robotics and Control with Active Uncertainty Reduction
• Gaussian Processes Model-based Control of Underactuated Balance Robots
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
• Continuous Deep Q-Learning with Simulator for Stabilization of Uncertain Discrete-Time Systems
• Bayesian Optimization Assisted Meal Bolus Decision Based on Gaussian Processes Learning and Risk-Sensitive Control
• Discriminator Augmented Model-Based Reinforcement Learning
• Gaussian Processes on Hypergraphs
• Composite Gaussian Processes: Scalable Computation and Performance Analysis
• A Hybrid PEM-GP Framework for Uncertainty-Aware System Identification of Quadcopters
• ILeSiA: Interactive Learning of Robot Situational Awareness from Camera Input
• Predicting Sim-to-Real Transfer with Probabilistic Dynamics Models
• Lipschitz Optimisation for Lipschitz Interpolation
• Combining Model-Free Q-Ensembles and Model-Based Approaches for Informed Exploration
• A Hybrid Approach for Trajectory Control Design
• A Two-Part Controller Synthesis Approach for Nonlinear Stochastic Systems Perturbed by Lévy Noise Using Renewal Theory and HJB-Based Impulse Control