How more data can hurt: Instability and regularization in next-generation reservoir computing
arXiv:2407.08641 · doi:10.1063/5.0262977
Abstract
It has been found recently that more data can, counter-intuitively, hurt the performance of deep neural networks. Here, we show that a more extreme version of the phenomenon occurs in data-driven models of dynamical systems. To elucidate the underlying mechanism, we focus on next-generation reservoir computing (NGRC) -- a popular framework for learning dynamics from data. We find that, despite learning a better representation of the flow map with more training data, NGRC can adopt an ill-conditioned ``integrator'' and lose stability. We link this data-induced instability to the auxiliary dimensions created by the delayed states in NGRC. Based on these findings, we propose simple strategies to mitigate the instability, either by increasing regularization strength in tandem with data size, or by carefully introducing noise during training. Our results highlight the importance of proper regularization in data-driven modeling of dynamical systems.
arXiv commentPublished version
References in corpus (35)
- Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
- Reconciling modern machine learning practice and the bias-variance trade-off
- Fourier Neural Operator for Parametric Partial Differential Equations
- Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data
- Next Generation Reservoir Computing
- Neural Ordinary Differential Equations
- Attractor Reconstruction by Machine Learning
- Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model
- On Explaining the Surprising Success of Reservoir Computing Forecaster of Chaos? The Universal Machine Learning Dynamical System with Contrasts to VAR and DMD
- Two models of double descent for weak features
- Network Structure Effects in Reservoir Computers
- Anticipating synchronization with machine learning
- Do Reservoir Computers Work Best at the Edge of Chaos?
- Doubly Transient Chaos: The Generic Form of Chaos in Autonomous Dissipative Systems
- Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations
- The Size of the Sync Basin Revisited
- Optimal Regularization Can Mitigate Double Descent
- Learning Spatiotemporal Chaos Using Next-Generation Reservoir Computing
- Multifunctionality in a Reservoir Computer
- More Data Can Hurt for Linear Regression: Sample-wise Double Descent
- Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components
- Deeper but smaller: Higher-order interactions increase linear stability but shrink basins
- Deep reconstruction of strange attractors from time series
- Learning unseen coexisting attractors
- Hypergraph reconstruction from dynamics
- Controlling Chaos Using Edge Computing Hardware
- Catch-22s of reservoir computing
- Data-driven discovery and extrapolation of parameterized pattern-forming dynamics
- Controlling Chaotic Maps using Next-Generation Reservoir Computing
- Hybridizing Traditional and Next-Generation Reservoir Computing to Accurately and Efficiently Forecast Dynamical Systems
- Symmetry-Aware Reservoir Computing
- Recovering sparse networks: Basis adaptation and stability under extensions
- Differentiable reservoir computing
- Out-of-Domain Generalization in Dynamical Systems Reconstruction
- Unifying Grokking and Double Descent