Neural Optimization Kernel: Towards Robust Deep Learning

Abstract

Deep neural networks (NN) have achieved great success in many applications. However, why do deep neural networks obtain good generalization at an over-parameterization regime is still unclear. To better understand deep NN, we establish the connection between deep NN and a novel kernel family, i.e., Neural Optimization Kernel (NOK). The architecture of structured approximation of NOK performs monotonic descent updates of implicit regularization problems. We can implicitly choose the regularization problems by employing different activation functions, e.g., ReLU, max pooling, and soft-thresholding. We further establish a new generalization bound of our deep structured approximated NOK architecture. Our unsupervised structured approximated NOK block can serve as a simple plug-in of popular backbones for a good generalization against input noise.

arXiv commentDeep Learning, Kernel Methods, Deep Learning Theory, Kernel Approximation, Integral Approximation

References in corpus (14)

• Playing Atari with Deep Reinforcement Learning
• Nearly unbiased variable selection under minimax concave penalty
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Wide Residual Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Masked Autoencoders Are Scalable Vision Learners
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Neural Kernels Without Tangents
• Theoretical properties of the global optimizer of two layer neural network
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime