Generalisation in fully-connected neural networks for time series forecasting
arXiv:1902.05312
Abstract
In this paper we study the generalization capabilities of fully-connected neural networks trained in the context of time series forecasting. Time series do not satisfy the typical assumption in statistical learning theory of the data being i.i.d. samples from some data-generating distribution. We use the input and weight Hessians, that is the smoothness of the learned function with respect to the input and the width of the minimum in weight space, to quantify a network's ability to generalize to unseen data. While such generalization metrics have been studied extensively in the i.i.d. setting of for example image recognition, here we empirically validate their use in the task of time series forecasting. Furthermore we discuss how one can control the generalization capability of the network by means of the training process using the learning rate, batch size and the number of training iterations as controls. Using these hyperparameters one can efficiently control the complexity of the output function without imposing explicit constraints.
References in corpus (14)
- Understanding deep learning requires rethinking generalization
- Opening the Black Box of Deep Neural Networks via Information
- The Loss Surfaces of Multilayer Networks
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Generalisation in humans and deep neural networks
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Robust Large Margin Deep Neural Networks
- Three Factors Influencing Minima in SGD
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- The statistics of critical points of Gaussian fields on large-dimensional spaces
- Stochastic Gradient Descent as Approximate Bayesian Inference
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Geometry of energy landscapes and the optimizability of deep neural networks
- Theory and Algorithms for Forecasting Time Series