PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNs

Abstract

In this paper, we derive a PAC-Bayes bound on the generalisation gap, in a supervised time-series setting for a special class of discrete-time non-linear dynamical systems. This class includes stable recurrent neural networks (RNN), and the motivation for this work was its application to RNNs. In order to achieve the results, we impose some stability constraints, on the allowed models. Here, stability is understood in the sense of dynamical systems. For RNNs, these stability conditions can be expressed in terms of conditions on the weights. We assume the processes involved are essentially bounded and the loss functions are Lipschitz. The proposed bound on the generalisation gap depends on the mixing coefficient of the data distribution, and the essential supremum of the data. Furthermore, the bound converges to zero as the dataset size increases. In this paper, we 1) formalize the learning problem, 2) derive a PAC-Bayesian error bound for such systems, 3) discuss various consequences of this error bound, and 4) show an illustrative example, with discussions on computing the proposed bound. Unlike other available bounds the derived bound holds for non i.i.d. data (time-series) and it does not grow with the number of steps of the RNN.

arXiv commentAccepted to AAAI2024 conference

References in corpus (22)

• Distilling the Knowledge in a Neural Network
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• On the properties of variational approximations of Gibbs posteriors
• Logarithmic Regret Bound in Partially Observable Linear Dynamical Systems
• Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification
• Learning Linear Dynamical Systems with Semi-Parametric Least Squares
• Active Learning for Nonlinear System Identification with Guarantees
• Learning nonlinear dynamical systems from a single trajectory
• On Generalization Bounds of a Family of Recurrent Neural Networks
• Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization
• Spectral Filtering for General Linear Dynamical Systems
• Non-asymptotic and Accurate Learning of Nonlinear Dynamical Systems
• Finite-sample analysis of identification of switched linear systems with arbitrary or restricted switching
• Transformers as Algorithms: Generalization and Stability in In-context Learning
• Single Trajectory Nonparametric Learning of Nonlinear Dynamics
• On Empirical Risk Minimization with Dependent and Heavy-Tailed Data
• PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNs
• Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets
• Explicit construction of the minimum error variance estimator for stochastic LTI state-space systems
• FLEX: an Adaptive Exploration Algorithm for Nonlinear Systems
• PAC-Bayesian bounds for learning LTI-ss systems with input from empirical loss
• Finite Sample Identification of Bilinear Dynamical Systems

Cited by in corpus (1)

• PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNs