Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint

Abstract

Motivated by the gap between theoretical optimal approximation rates of deep neural networks (DNNs) and the accuracy realized in practice, we seek to improve the training of DNNs. The adoption of an adaptive basis viewpoint of DNNs leads to novel initializations and a hybrid least squares/gradient descent optimizer. We provide analysis of these techniques and illustrate via numerical examples dramatic increases in accuracy and convergence rate for benchmarks characterizing scientific applications where DNNs are currently used, including regression problems and physics-informed neural networks for the solution of partial differential equations.

arXiv comment26 pages

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Representation Benefits of Deep Feedforward Networks
• Understanding Deep Neural Networks with Rectified Linear Units
• Stochastic Conditional Generative Networks with Basis Decomposition

Cited by in corpus (6)

• hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition
• Data-driven learning of robust nonlocal physics from high-fidelity synthetic data
• The gap between theory and practice in function approximation with deep neural networks
• Partition of unity networks: deep hp-approximation
• Galerkin Neural Networks: A Framework for Approximating Variational Equations with Error Control
• Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance