Information Theoretic Regret Bounds for Online Nonlinear Control

Abstract

This work studies the problem of sequential control in an unknown, nonlinear dynamical system, where we model the underlying system dynamics as an unknown function in a known Reproducing Kernel Hilbert Space. This framework yields a general setting that permits discrete and continuous control inputs as well as non-smooth, non-differentiable dynamics. Our main result, the Lower Confidence-based Continuous Control ($L{C}^3$) algorithm, enjoys a near-optimal $O(\sqrt{T})$ regret bound against the optimal controller in episodic settings, where $T$ is the number of episodes. The bound has no explicit dependence on dimension of the system dynamics, which could be infinite, but instead only depends on information theoretic quantities. We empirically show its application to a number of nonlinear control tasks and demonstrate the benefit of exploration for learning model dynamics.

References in corpus (18)

• Solving Rubik's Cube with a Robot Hand
• Model-Ensemble Trust-Region Policy Optimization
• Exploration by Random Network Distillation
• Benchmarking Model-Based Reinforcement Learning
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• Exploring Model-based Planning with Policy Networks
• Model-Based Reinforcement Learning with Value-Targeted Regression
• Agnostic System Identification for Model-Based Reinforcement Learning
• Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Logarithmic Regret for Online Control
• Naive Exploration is Optimal for Online LQR
• Active Learning for Nonlinear System Identification with Guarantees
• Information-Theoretic Confidence Bounds for Reinforcement Learning
• The Nonstochastic Control Problem
• Lyceum: An efficient and scalable ecosystem for robot learning

Cited by in corpus (22)

• Bilinear Classes: A Structural Framework for Provable Generalization in RL
• Meta-Adaptive Nonlinear Control: Theory and Algorithms
• On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spaces
• Robust Policy Gradient against Strong Data Corruption
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
• On Information Gain and Regret Bounds in Gaussian Process Bandits
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature
• Representation Learning for Online and Offline RL in Low-rank MDPs
• MobILE: Model-Based Imitation Learning From Observation Alone
• Online Robust Control of Nonlinear Systems with Large Uncertainty
• Provable Regret Bounds for Deep Online Learning and Control
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
• Towards a Dimension-Free Understanding of Adaptive Linear Control
• Task-Optimal Exploration in Linear Dynamical Systems
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Koopman Spectrum Nonlinear Regulators and Efficient Online Learning
• Sparsity in Partially Controllable Linear Systems
• Robust Online Control with Model Misspecification
• Effects of sampling and horizon in predictive reinforcement learning