Koopman Spectrum Nonlinear Regulators and Efficient Online Learning

Abstract

Most modern reinforcement learning algorithms optimize a cumulative single-step cost along a trajectory. The optimized motions are often 'unnatural', representing, for example, behaviors with sudden accelerations that waste energy and lack predictability. In this work, we present a novel paradigm of controlling nonlinear systems via the minimization of the Koopman spectrum cost: a cost over the Koopman operator of the controlled dynamics. This induces a broader class of dynamical behaviors that evolve over stable manifolds such as nonlinear oscillators, closed loops, and smooth movements. We demonstrate that some dynamics characterizations that are not possible with a cumulative cost are feasible in this paradigm, which generalizes the classical eigenstructure and pole assignments to nonlinear decision making. Moreover, we present a sample efficient online learning algorithm for our problem that enjoys a sub-linear regret bound under some structural assumptions.

41 pages, 21 figures

References in corpus (12)

• Playing Atari with Deep Reinforcement Learning
• Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• DeepMind Control Suite
• Phase reduction approach to synchronization of nonlinear oscillators
• Isostables, isochrons, and Koopman spectrum for the action-angle representation of stable fixed point dynamics
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
• Operator-valued Kernels for Learning from Functional Response Data
• Learning Compositional Koopman Operators for Model-Based Control
• Active Learning for Nonlinear System Identification with Guarantees
• Information Theoretic Regret Bounds for Online Nonlinear Control