A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks

Abstract

To investigate neural network parameters, it is easier to study the distribution of parameters than to study the parameters in each neuron. The ridgelet transform is a pseudo-inverse operator that maps a given function $f$ to the parameter distribution $\gamma$ so that a network $\mathtt{NN}[\gamma ]$ reproduces $f$, i.e. $\mathtt{NN}[\gamma ]=f$. For depth-2 fully-connected networks on a Euclidean space, the ridgelet transform has been discovered up to the closed-form expression, thus we could describe how the parameters are distributed. However, for a variety of modern neural network architectures, the closed-form expression has not been known. In this paper, we explain a systematic method using Fourier expressions to derive ridgelet transforms for a variety of modern networks such as networks on finite fields ${\mathbb{F}}_p$, group convolutional networks on abstract Hilbert space $\mathcal{H}$, fully-connected networks on noncompact symmetric spaces $G/K$, and pooling layers, or the $d$-plane ridgelet transform.

References in corpus (5)

• Hyperbolic Geometry of Complex Networks
• A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case
• How do infinite width bounded norm networks look in function space?
• Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networks
• Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks