A Kernel Perspective for Regularizing Deep Neural Networks

Abstract

We propose a new point of view for regularizing deep neural networks by using the norm of a reproducing kernel Hilbert space (RKHS). Even though this norm cannot be computed, it admits upper and lower approximations leading to various practical strategies. Specifically, this perspective (i) provides a common umbrella for many existing regularization principles, including spectral norm and gradient penalties, or adversarial training, (ii) leads to new effective regularization penalties, and (iii) suggests hybrid strategies combining lower and upper bounds to get better approximations of the RKHS norm. We experimentally show this approach to be effective when learning on small datasets, or to obtain adversarially robust models.

arXiv commentICML

Cited by in corpus (15)

• Image Classification with Small Datasets: Overview and Benchmark
• Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows
• Distributionally Robust Optimization and Generalization in Kernel Methods
• Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification
• Kernel Distributionally Robust Optimization
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Framing RNN as a kernel method: A neural ODE approach
• Generalised Lipschitz Regularisation Equals Distributional Robustness
• Input Hessian Regularization of Neural Networks
• Smoothed Inference for Adversarially-Trained Models
• Adversarial training applied to Convolutional Neural Network for photometric redshift predictions
• Convolutional Spectral Kernel Learning
• Distributional Robustness Regularized Scenario Optimization with Application to Model Predictive Control
• Adversarially Robust Kernel Smoothing
• On Inductive Biases for Machine Learning in Data Constrained Settings