Neural Optimization Kernel: Towards Robust Deep Learning

Abstract

Deep neural networks (NN) have achieved great success in many applications. However, why do deep neural networks obtain good generalization at an over-parameterization regime is still unclear. To better understand deep NN, we establish the connection between deep NN and a novel kernel family, i.e., Neural Optimization Kernel (NOK). The architecture of structured approximation of NOK performs monotonic descent updates of implicit regularization problems. We can implicitly choose the regularization problems by employing different activation functions, e.g., ReLU, max pooling, and soft-thresholding. We further establish a new generalization bound of our deep structured approximated NOK architecture. Our unsupervised structured approximated NOK block can serve as a simple plug-in of popular backbones for a good generalization against input noise.

Deep Learning, Kernel Methods, Deep Learning Theory, Kernel Approximation, Integral Approximation

References in corpus (5)

• Nearly unbiased variable selection under minimax concave penalty
• Masked Autoencoders Are Scalable Vision Learners
• Neural Kernels Without Tangents
• Theoretical properties of the global optimizer of two layer neural network
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime