Reproducing Activation Function for Deep Learning

Abstract

We propose reproducing activation functions (RAFs) to improve deep learning accuracy for various applications ranging from computer vision to scientific computing. The idea is to employ several basic functions and their learnable linear combination to construct neuron-wise data-driven activation functions for each neuron. Armed with RAFs, neural networks (NNs) can reproduce traditional approximation tools and, therefore, approximate target functions with a smaller number of parameters than traditional NNs. In NN training, RAFs can generate neural tangent kernels (NTKs) with a better condition number than traditional activation functions lessening the spectral bias of deep learning. As demonstrated by extensive numerical tests, the proposed RAFs can facilitate the convergence of deep learning optimization for a solution with higher accuracy than existing deep learning solvers for audio/image/video reconstruction, PDEs, and eigenvalue problems. With RAFs, the errors of audio/video reconstruction, PDEs, and eigenvalue problems are decreased by over 14%, 73%, 99%, respectively, compared with baseline, while the performance of image reconstruction increases by 58%.

References in corpus (36)

• Adam: A Method for Stochastic Optimization
• Solving high-dimensional partial differential equations using deep learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Deep Network Approximation for Smooth Functions
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
• Convergence of the Deep BSDE Method for Coupled FBSDEs
• Deep Nitsche Method: Deep Ritz Method with Essential Boundary Conditions
• Neural Network Approximation: Three Hidden Layers Are Enough
• When and why PINNs fail to train: A neural tangent kernel perspective
• Deep least-squares methods: an unsupervised learning-based numerical method for solving elliptic PDEs
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approach
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Theory of the Frequency Principle for General Deep Neural Networks
• SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations
• Int-Deep: A Deep Learning Initialized Iterative Method for Nonlinear Problems
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Multi-scale Deep Neural Network (MscaleDNN) Methods for Oscillatory Stokes Flows in Complex Domains
• The phase diagram of approximation rates for deep neural networks
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• A Priori Estimates of the Population Risk for Residual Networks
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Structure Probing Neural Network Deflation
• Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
• Enforcing exact boundary and initial conditions in the deep mixed residual method
• Implicit Regularization of Stochastic Gradient Descent in Natural Language Processing: Observations and Implications
• Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent
• Quasi-Monte Carlo sampling for machine-learning partial differential equations

Cited by in corpus (5)

• Solving PDEs on Unknown Manifolds with Machine Learning
• A Deep Learning Based Discontinuous Galerkin Method for Hyperbolic Equations with Discontinuous Solutions and Random Uncertainties
• Adaptive Learning on the Grids for Elliptic Hemivariational Inequalities
• Multiscale and Nonlocal Learning for PDEs using Densely Connected RNNs
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network