Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks

Abstract

Deep networks, especially convolutional neural networks (CNNs), have been successfully applied in various areas of machine learning as well as to challenging problems in other scientific and engineering fields. This paper introduces Butterfly-Net, a low-complexity CNN with structured and sparse cross-channel connections, together with a Butterfly initialization strategy for a family of networks. Theoretical analysis of the approximation power of Butterfly-Net to the Fourier representation of input data shows that the error decays exponentially as the depth increases. Combining Butterfly-Net with a fully connected neural network, a large class of problems are proved to be well approximated with network complexity depending on the effective frequency bandwidth instead of the input dimension. Regular CNN is covered as a special case in our analysis. Numerical experiments validate the analytical results on the approximation of Fourier kernels and energy functionals of Poisson's equations. Moreover, all experiments support that training from Butterfly initialization outperforms training from random initialization. Also, adding the remaining cross-channel connections, although significantly increase the parameter number, does not much improve the post-training accuracy and is more sensitive to data distribution.

References in corpus (10)

• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• On the Number of Linear Regions of Deep Neural Networks
• A unified deep artificial neural network approach to partial differential equations in complex geometries
• Learning Functions: When Is Deep Better Than Shallow
• Benefits of depth in neural networks
• BCR-Net: a neural network based on the nonstandard wavelet form
• Variational training of neural network approximations of solution maps for physical models
• Interpolative Butterfly Factorization
• A multiscale neural network based on hierarchical matrices
• Multidimensional Butterfly Factorization

Cited by in corpus (10)

• VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images
• BCR-Net: a neural network based on the nonstandard wavelet form
• Variational training of neural network approximations of solution maps for physical models
• A multiscale neural network based on hierarchical matrices
• SwitchNet: a neural network model for forward and inverse scattering problems
• Solving the Wide-band Inverse Scattering Problem via Equivariant Neural Networks
• Sparse Linear Networks with a Fixed Butterfly Structure: Theory and Practice
• A reduced order Schwarz method for nonlinear multiscale elliptic equations based on two-layer neural networks
• Accurate and Robust Deep Learning Framework for Solving Wave-Based Inverse Problems in the Super-Resolution Regime
• A multiscale neural network based on hierarchical nested bases