From deep to Shallow: Equivalent Forms of Deep Networks in Reproducing Kernel Krein Space and Indefinite Support Vector Machines

Abstract

In this paper we explore a connection between deep networks and learning in reproducing kernel Krein space. Our approach is based on the concept of push-forward - that is, taking a fixed non-linear transform on a linear projection and converting it to a linear projection on the output of a fixed non-linear transform, pushing the weights forward through the non-linearity. Applying this repeatedly from the input to the output of a deep network, the weights can be progressively "pushed" to the output layer, resulting in a flat network that has the form of a fixed non-linear map (whose form is determined by the structure of the deep network) followed by a linear projection determined by the weight matrices - that is, we take a deep network and convert it to an equivalent (indefinite) kernel machine. We then investigate the implications of this transformation for capacity control and uniform convergence, and provide a Rademacher complexity bound on the deep network in terms of Rademacher complexity in reproducing kernel Krein space. Finally, we analyse the sparsity properties of the flat representation, showing that the flat weights are (effectively) Lp-"norm" regularised with 0<p<1 (bridge regression).

References in corpus (14)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Spectrally-normalized margin bounds for neural networks
• Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Uniform convergence may be unable to explain generalization in deep learning
• Sparse DNNs with Improved Adversarial Robustness
• SGD Learns the Conjugate Kernel Class of the Network
• Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
• Semi-indefinite-inner-product and generalized Minkowski spaces
• Scalable Learning in Reproducing Kernel Krein Spaces
• Learning the Kernel for Classification and Regression