Orthogonal Transforms in Neural Networks Amount to Effective Regularization

Abstract

We consider applications of neural networks in nonlinear system identification and formulate a hypothesis that adjusting general network structure by incorporating frequency information or other known orthogonal transform, should result in an efficient neural network retaining its universal properties. We show that such a structure is a universal approximator and that using any orthogonal transform in a proposed way implies regularization during training by adjusting the learning rate of each parameter individually. We empirically show in particular, that such a structure, using the Fourier transform, outperforms equivalent models without orthogonality support.

References in corpus (5)

• Recent Advances in Physical Reservoir Computing: A Review
• Fourier Neural Operator for Parametric Partial Differential Equations
• Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks
• Transformers for Modeling Physical Systems
• Nonlinear system modeling based on constrained Volterra series estimates