Towards Understanding Theoretical Advantages of Complex-Reaction Networks

Abstract

Complex-valued neural networks have attracted increasing attention in recent years, while it remains open on the advantages of complex-valued neural networks in comparison with real-valued networks. This work takes one step on this direction by introducing the \emph{complex-reaction network} with fully-connected feed-forward architecture. We prove the universal approximation property for complex-reaction networks, and show that a class of radial functions can be approximated by a complex-reaction network using the polynomial number of parameters, whereas real-valued networks need at least exponential parameters to reach the same approximation level. For empirical risk minimization, our theoretical result shows that the critical point set of complex-reaction networks is a proper subset of that of real-valued networks, which may show some insights on finding the optimal solutions more easily for complex-reaction networks.

References in corpus (18)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Complex Embeddings for Simple Link Prediction
• Understanding deep learning requires rethinking generalization
• A Convergence Theory for Deep Learning via Over-Parameterization
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Train faster, generalize better: Stability of stochastic gradient descent
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• The Power of Depth for Feedforward Neural Networks
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Associative Long Short-Term Memory
• A Survey of Complex-Valued Neural Networks
• Complex Gated Recurrent Neural Networks
• Evaluation of Complex-Valued Neural Networks on Real-Valued Classification Tasks
• Flexible Transmitter Network
• The universal approximation theorem for complex-valued neural networks

Cited by in corpus (5)

• Deep Complex-valued Radial Basis Function Neural Networks and Parameter Selection
• On the Parameter Selection of Phase-transmittance Radial Basis Function Neural Networks for Communication Systems
• On the Computational Complexities of Complex-valued Neural Networks
• CVNN-based Channel Estimation and Equalization in OFDM Systems Without Cyclic Prefix
• Theoretical Exploration of Flexible Transmitter Model