Neural Networks with Complex-Valued Weights Have No Spurious Local Minima
arXiv:2103.07287
Abstract
We study the benefits of complex-valued weights for neural networks. We prove that shallow complex neural networks with quadratic activations have no spurious local minima. In contrast, shallow real neural networks with quadratic activations have infinitely many spurious local minima under the same conditions. In addition, we provide specific examples to demonstrate that complex-valued weights turn poor local minima into saddle points.
References in corpus (18)
- The Complex Gradient Operator and the CR-Calculus
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- The loss surface of deep and wide neural networks
- On the Power of Over-parametrization in Neural Networks with Quadratic Activation
- Full-Capacity Unitary Recurrent Neural Networks
- Associative Long Short-Term Memory
- On Complex Valued Convolutional Neural Networks
- Phase-aware Speech Enhancement with Deep Complex U-Net
- The Global Landscape of Neural Networks: An Overview
- Complex-Valued Autoencoders
- On the loss landscape of a class of deep neural networks with no bad local valleys
- Complex Gated Recurrent Neural Networks
- On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
- Wirtinger's Calculus in general Hilbert Spaces
- Small nonlinearities in activation functions create bad local minima in neural networks
- Network size and weights size for memorization with two-layers neural networks
- Stationary Points of Shallow Neural Networks with Quadratic Activation Function
- Optimization Landscapes of Wide Deep Neural Networks Are Benign