Why Shallow Networks Struggle to Approximate and Learn High Frequencies

Abstract

In this work, we present a comprehensive study combining mathematical and computational analysis to explain why a two-layer neural network struggles to handle high frequencies in both approximation and learning, especially when machine precision, numerical noise, and computational cost are significant factors in practice. Specifically, we investigate the following fundamental computational issues: (1) the minimal numerical error achievable under finite precision, (2) the computational cost required to attain a given accuracy, and (3) the stability of the method with respect to perturbations. The core of our analysis lies in the conditioning of the representation and its learning dynamics. Explicit answers to these questions are provided, along with supporting numerical evidence.

References in corpus (12)

• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Neural Network Approximation: Three Hidden Layers Are Enough
• Deep Network Approximation Characterized by Number of Neurons
• Nonlinear Approximation via Compositions
• On the Existence of Simpler Machine Learning Models
• Theory of the Frequency Principle for General Deep Neural Networks
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach
• Conditioning of Finite Element Equations with Arbitrary Anisotropic Meshes
• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• Training Two-Layer ReLU Networks with Gradient Descent is Inconsistent