Neural network with optimal neuron activation functions based on additive Gaussian process regression

Abstract

Feed-forward neural networks (NN) are a staple machine learning method widely used in many areas of science and technology. While even a single-hidden layer NN is a universal approximator, its expressive power is limited by the use of simple neuron activation functions (such as sigmoid functions) that are typically the same for all neurons. More flexible neuron activation functions would allow using fewer neurons and layers and thereby save computational cost and improve expressive power. We show that additive Gaussian process regression (GPR) can be used to construct optimal neuron activation functions that are individual to each neuron. An approach is also introduced that avoids non-linear fitting of neural network parameters. The resulting method combines the advantage of robustness of a linear regression with the higher expressive power of a NN. We demonstrate the approach by fitting the potential energy surfaces of the water molecule and formaldehyde. Without requiring any non-linear optimization, the additive GPR based approach outperforms a conventional NN in the high accuracy regime, where a conventional NN suffers more from overfitting.

References in corpus (5)

• SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects
• Permutationally invariant polynomial regression for energies and gradients, using reverse differentiation, achieves orders of magnitude speed-up with high precision compared to other machine learning methods
• On the optimization of hyperparameters in Gaussian process regression with the help of low-order high-dimensional model representation
• Easy representation of multivariate functions with low-dimensional terms via Gaussian process regression kernel design: applications to machine learning of potential energy surfaces and kinetic energy densities from sparse data
• REANN: A PyTorch-based End-to-End Multi-functional Deep Neural Network Package for Molecular, Reactive and Periodic Systems

Cited by in corpus (4)

• Degeneration of kernel regression with Matern kernels into low-order polynomial regression in high dimension
• Orders-of-coupling representation with a single neural network with optimal neuron activation functions and without nonlinear parameter optimization
• Machine learning the screening factor in the soft bond valence approach for rapid crystal structure estimation
• Machine learning-guided construction of an analytic kinetic energy functional for orbital free density functional theory